=== RUN TestOdhOperator/DSCInitialization_and_DataScienceCluster_validation_E2E_Tests/Validate_DSCI_webhook_denies_invalid_PEM_on_update testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/DSCInitialization_and_DataScienceCluster_validation_E2E_Tests/Validate_DSCI_webhook_denies_invalid_PEM_on_update (300.01s) === RUN TestOdhOperator/DSCInitialization_and_DataScienceCluster_validation_E2E_Tests 2026-07-08T13:45:17Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.19.18"}} test_context_test.go:1675: Timed out after 300.008s. Webhook should deny update with invalid PEM in customCABundle The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1664 with: Webhook should deny update with invalid PEM in customCABundle Expected an error to have occurred. Got: <nil>: nil --- FAIL: TestOdhOperator/DSCInitialization_and_DataScienceCluster_validation_E2E_Tests (300.28s) === RUN TestOdhOperator/components/group_1/aigateway/Validate_component_enabled 2026-07-08T13:50:28Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.19.18"}} testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_1/aigateway/Validate_component_enabled (300.00s) === RUN TestOdhOperator/components/group_1/aigateway === PAUSE TestOdhOperator/components/group_1/aigateway === CONT TestOdhOperator/components/group_1/aigateway resource_applier_test.go:209: Timed out after 300.001s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:121 with: Expected resource 'default-dsc' of kind 'DataScienceCluster' to match condition {.spec.components.aigateway.managementState == "Removed" []}. Expected <string>: map[apiVersion:datasciencecluster.opendatahub.io/v2 kind:DataScienceCluster metadata:map[annotations:map[kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"datasciencecluster.opendatahub.io/v2","kind":"DataScienceCluster","metadata":{"annotations":{},"name":"default-dsc"},"spec":{"components":{"aipipelines":{"managementState":"Removed"},"codeflare":{"managementState":"Removed"},"dashboard":{"managementState":"Managed"},"kserve":{"defaultDeploymentMode":"RawDeployment","managementState":"Managed","serving":{"managementState":"Removed","name":"knative-serving"}},"kueue":{"managementState":"Removed"},"llamastackoperator":{"managementState":"Removed"},"mlflowoperator":{"managementState":"Removed"},"modelregistry":{"managementState":"Removed"},"ogx":{"managementState":"Removed"},"ray":{"managementState":"Removed"},"sparkoperator":{"managementState":"Removed"},"trainer":{"managementState":"Removed"},"trainingoperator":{"managementState":"Removed"},"trustyai":{"managementState":"Removed"},"workbenches":{"managementState":"Removed"}}}} ] creationTimestamp:2026-06-29T13:43:26Z generation:1885 managedFields:[map[apiVersion:datasciencecluster.opendatahub.io/v2 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:kubectl.kubernetes.io/last-applied-configuration:map[]]] f:status:map[f:components:map[f:aipipelines:map[f:managementState:map[] f:releases:map[k:{"name":"Kubeflow Pipelines"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:dashboard:map[f:managementState:map[]] f:feastoperator:map[f:managementState:map[] f:releases:map[k:{"name":"Feast"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:kserve:map[f:managementState:map[] f:releases:map[k:{"name":"KServe"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-inference-scheduler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-workload-variant-autoscaler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"vLLM"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:kueue:map[f:managementState:map[]] f:llamastackoperator:map[] f:mlflowoperator:map[f:managementState:map[] f:releases:map[k:{"name":"MLflow"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:modelregistry:map[f:managementState:map[] f:registriesNamespace:map[] f:releases:map[k:{"name":"Kubeflow Model Registry"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"Open Data Hub Model Registry Operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:ogx:map[f:managementState:map[] f:releases:map[k:{"name":"OGX Operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"OGX"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:ray:map[f:releases:map[k:{"name":"KubeRay"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:sparkoperator:map[f:managementState:map[] f:releases:map[k:{"name":"Spark Operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"Spark"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:trainer:map[f:managementState:map[] f:releases:map[k:{"name":"Kubeflow Trainer"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:trainingoperator:map[f:managementState:map[] f:releases:map[k:{"name":"Kubeflow Training Operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:trustyai:map[f:managementState:map[] f:releases:map[k:{"name":"EvalHub"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI Guardrails orchestrator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI LMEval driver"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI LMEval job"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI builtin detectors"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI service"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI sidecar gateway"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:workbenches:map[f:managementState:map[] f:releases:map[k:{"name":"Kubeflow Notebook Controller"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]] f:workbenchNamespace:map[]]] f:observedGeneration:map[] f:release:map[f:name:map[] f:version:map[]]]] manager:modules operation:Apply subresource:status time:2026-07-08T13:53:57Z] map[apiVersion:datasciencecluster.opendatahub.io/v2 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:kubectl.kubernetes.io/last-applied-configuration:map[]]] f:status:map[f:components:map[f:aipipelines:map[f:managementState:map[]] f:dashboard:map[f:managementState:map[]] f:feastoperator:map[f:managementState:map[]] f:kserve:map[f:managementState:map[] f:releases:map[k:{"name":"KServe"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-inference-scheduler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-workload-variant-autoscaler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"vLLM"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:kueue:map[f:managementState:map[]] f:llamastackoperator:map[] f:mlflowoperator:map[f:managementState:map[] f:releases:map[k:{"name":"MLflow"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:modelregistry:map[f:managementState:map[]] f:ogx:map[f:managementState:map[]] f:ray:map[f:managementState:map[]] f:sparkoperator:map[f:managementState:map[] f:releases:map[k:{"name":"Spark Operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"Spark"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:trainer:map[f:managementState:map[]] f:trainingoperator:map[f:managementState:map[]] f:trustyai:map[f:managementState:map[] f:releases:map[k:{"name":"EvalHub"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI Guardrails orchestrator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI LMEval driver"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI LMEval job"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI builtin detectors"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI service"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI sidecar gateway"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:workbenches:map[f:managementState:map[]]] f:conditions:map[] f:observedGeneration:map[] f:phase:map[] f:release:map[f:name:map[] f:version:map[]]]] manager:datasciencecluster operation:Apply subresource:status time:2026-07-08T13:54:40Z] map[apiVersion:datasciencecluster.opendatahub.io/v2 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[.:map[] f:kubectl.kubernetes.io/last-applied-configuration:map[]]] f:spec:map[.:map[] f:components:map[.:map[] f:aipipelines:map[] f:dashboard:map[] f:kserve:map[.:map[] f:rawDeploymentServiceConfig:map[]] f:kueue:map[.:map[] f:defaultClusterQueueName:map[] f:defaultLocalQueueName:map[]] f:llamastackoperator:map[] f:mlflowoperator:map[] f:modelregistry:map[.:map[] f:registriesNamespace:map[]] f:ogx:map[] f:ray:map[] f:sparkoperator:map[] f:trainer:map[] f:trainingoperator:map[] f:trustyai:map[.:map[] f:mcpGuardrailsMode:map[]] f:workbenches:map[.:map[] f:workbenchNamespace:map[]]]]] manager:kubectl-client-side-apply operation:Update time:2026-06-29T13:43:26Z] map[apiVersion:datasciencecluster.opendatahub.io/v2 fieldsType:FieldsV1 fieldsV1:map[f:spec:map[f:components:map[f:kserve:map[f:modelsAsService:map[.:map[] f:managementState:map[]]] f:llamastackoperator:map[f:managementState:map[]] f:mlflowoperator:map[f:managementState:map[]] f:trustyai:map[f:managementState:map[]]]]] manager:kubectl-patch operation:Update time:2026-07-08T13:00:17Z] map[apiVersion:datasciencecluster.opendatahub.io/v2 fieldsType:FieldsV1 fieldsV1:map[f:spec:map[f:components:map[f:aipipelines:map[f:managementState:map[]] f:dashboard:map[f:managementState:map[]] f:feastoperator:map[.:map[] f:managementState:map[]] f:kserve:map[f:managementState:map[] f:wva:map[.:map[] f:managementState:map[]]] f:kueue:map[f:managementState:map[]] f:modelregistry:map[f:managementState:map[]] f:ogx:map[f:managementState:map[]] f:ray:map[f:managementState:map[]] f:sparkoperator:map[f:managementState:map[]] f:trainer:map[f:managementState:map[]] f:trainingoperator:map[f:managementState:map[]] f:workbenches:map[f:managementState:map[]]]]] manager:e2e-tests operation:Update time:2026-07-08T13:53:57Z]] name:default-dsc resourceVersion:384279237 uid:191cdb15-01aa-4da4-9798-e6b508adbf9f] spec:map[components:map[aipipelines:map[managementState:Removed] dashboard:map[managementState:Managed] feastoperator:map[managementState:Removed] kserve:map[managementState:Managed modelsAsService:map[managementState:Managed] rawDeploymentServiceConfig:Headless wva:map[managementState:Removed]] kueue:map[defaultClusterQueueName:default defaultLocalQueueName:default managementState:Unmanaged] llamastackoperator:map[managementState:Removed] mlflowoperator:map[managementState:Managed] modelregistry:map[managementState:Removed registriesNamespace:rhoai-model-registries] ogx:map[managementState:Removed] ray:map[managementState:Removed] sparkoperator:map[managementState:Managed] trainer:map[managementState:Removed] trainingoperator:map[managementState:Removed] trustyai:map[managementState:Managed mcpGuardrailsMode:false] workbenches:map[managementState:Removed workbenchNamespace:rhods-notebooks]]] status:map[components:map[aipipelines:map[managementState:Removed releases:[map[name:Kubeflow Pipelines repoUrl:https://github.com/kubeflow/pipelines version:2.16.0]]] dashboard:map[managementState:Managed] feastoperator:map[managementState:Removed releases:[map[name:Feast repoUrl:https://github.com/feast-dev/feast version:0.63.0]]] kserve:map[managementState:Managed releases:[map[name:KServe repoUrl:https://github.com/kserve/kserve/ version:v0.17.0] map[name:vLLM repoUrl:https://github.com/vllm-project/vllm version:v0.18.0] map[name:llm-d-inference-scheduler repoUrl:https://github.com/llm-d/llm-d-inference-scheduler version:v0.7.1] map[name:llm-d-workload-variant-autoscaler repoUrl:https://github.com/llm-d/llm-d-workload-variant-autoscaler version:v0.6.0]]] kueue:map[managementState:Unmanaged] llamastackoperator:map[] mlflowoperator:map[managementState:Managed releases:[map[name:MLflow repoUrl:https://github.com/mlflow/mlflow version:v3.12.0]]] modelregistry:map[managementState:Removed registriesNamespace:rhoai-model-registries releases:[map[name:Kubeflow Model Registry repoUrl:https://github.com/kubeflow/model-registry version:latest] map[name:Open Data Hub Model Registry Operator repoUrl:https://github.com/opendatahub-io/model-registry-operator version:latest]]] ogx:map[managementState:Removed releases:[map[name:OGX repoUrl:https://github.com/opendatahub-io/ogx-distribution version:v1.0.2] map[name:OGX Operator repoUrl:https://github.com/opendatahub-io/ogx-k8s-operator version:v0.10.0]]] ray:map[managementState:Removed releases:[map[name:KubeRay repoUrl:https://github.com/ray-project/kuberay version:1.4.2]]] sparkoperator:map[managementState:Managed releases:[map[name:Spark Operator repoUrl:https://github.com/opendatahub-io/spark-operator version:v2.4.0] map[name:Spark repoUrl:https://github.com/apache/spark version:4.0.1]]] trainer:map[managementState:Removed releases:[map[name:Kubeflow Trainer repoUrl:https://github.com/kubeflow/trainer version:2.1.0]]] trainingoperator:map[managementState:Removed releases:[map[name:Kubeflow Training Operator repoUrl:https://github.com/kubeflow/trainer version:1.9.0]]] trustyai:map[managementState:Managed releases:[map[name:TrustyAI operator repoUrl:https://github.com/trustyai-explainability/trustyai-service-operator version:v1.37.0] map[name:TrustyAI service repoUrl:https://github.com/trustyai-explainability/trustyai-explainability version:v0.28.0] map[name:TrustyAI LMEval driver repoUrl:https://github.com/trustyai-explainability/trustyai-service-operator version:v1.37.0] map[name:TrustyAI LMEval job repoUrl:https://github.com/EleutherAI/lm-evaluation-harness version:v0.4.8] map[name:TrustyAI Guardrails orchestrator repoUrl:https://github.com/foundation-model-stack/fms-guardrails-orchestrator version:0.9.4] map[name:TrustyAI builtin detectors repoUrl:https://github.com/trustyai-explainability/guardrails-regex-detector version:v0.2.0] map[name:TrustyAI sidecar gateway repoUrl:https://github.com/trustyai-explainability/vllm-orchestrator-gateway version:v0.2.1] map[name:EvalHub repoUrl:https://github.com/eval-hub/eval-hub version:v0.3.0]]] workbenches:map[managementState:Removed releases:[map[name:Kubeflow Notebook Controller repoUrl:https://github.com/kubeflow/kubeflow version:1.10.0]] workbenchNamespace:rhods-notebooks]] conditions:[map[lastTransitionTime:2026-07-08T13:54:39Z message:Some components are not ready: dashboard reason:NotReady status:False type:Ready] map[lastTransitionTime:2026-07-08T13:53:59Z observedGeneration:1885 status:True type:ProvisioningSucceeded] map[lastTransitionTime:2026-07-08T13:54:39Z message:Some components are not ready: dashboard reason:NotReady status:False type:ComponentsReady] map[lastTransitionTime:2026-07-08T13:54:40Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:AIPipelinesReady] map[lastTransitionTime:2026-07-08T13:54:40Z message:0/1 deployments ready reason:DeploymentsNotReady status:False type:DashboardReady] map[lastTransitionTime:2026-07-08T13:54:40Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:FeastOperatorReady] map[lastTransitionTime:2026-07-08T13:51:15Z status:True type:KserveLLMInferenceServiceDependencies] map[lastTransitionTime:2026-07-08T13:51:15Z status:True type:KserveLLMInferenceServiceWideEPDependencies] map[lastTransitionTime:2026-07-08T13:54:40Z status:True type:KserveReady] map[lastTransitionTime:2026-07-08T13:54:40Z status:True type:KueueReady] map[lastTransitionTime:2026-07-08T13:54:40Z status:True type:MLflowOperatorReady] map[lastTransitionTime:2026-07-08T13:50:39Z message:MaaS gateway annotations and Authorino TLS are correctly configured reason:MaaSPrerequisitesMet status:True type:MaaSPrerequisitesAvailable] map[lastTransitionTime:2026-07-08T13:54:40Z status:True type:ModelControllerReady] map[lastTransitionTime:2026-07-08T13:54:40Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:ModelRegistryReady] map[lastTransitionTime:2026-07-08T13:54:40Z message:MaaS platform manifests applied and maas-api deployment is available reason:Reconciled status:True type:ModelsAsServiceReady] map[lastTransitionTime:2026-07-08T13:54:40Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:OGXReady] map[lastTransitionTime:2026-07-08T13:54:40Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:RayReady] map[lastTransitionTime:2026-07-08T13:54:40Z status:True type:SparkOperatorReady] map[lastTransitionTime:2026-07-08T13:54:40Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:TrainerReady] map[lastTransitionTime:2026-07-08T13:54:40Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:TrainingOperatorReady] map[lastTransitionTime:2026-07-08T13:54:40Z status:True type:TrustyAIReady] map[lastTransitionTime:2026-07-08T13:54:40Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:WorkbenchesReady]] observedGeneration:1885 phase:Not Ready release:map[name:OpenShift AI Self-Managed version:3.5.0-ea.2]]] to match expression <string>: .spec.components.aigateway.managementState == "Removed" --- FAIL: TestOdhOperator/components/group_1/aigateway (300.54s) === RUN TestOdhOperator/components/group_1/kserve/Validate_component_spec testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_1/kserve/Validate_component_spec (600.03s) === RUN TestOdhOperator/components/group_1/kserve === PAUSE TestOdhOperator/components/group_1/kserve === CONT TestOdhOperator/components/group_1/kserve test_context_test.go:223: Timed out after 600.008s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:121 with: Expected resource 'default-kserve' of kind 'Kserve' to match condition {.spec.nim.managementState == "" []}. Expected <string>: map[apiVersion:components.platform.opendatahub.io/v1alpha1 kind:Kserve metadata:map[annotations:map[component.opendatahub.io/management-state:Managed platform.opendatahub.io/instance.generation:1887 platform.opendatahub.io/instance.name:default-dsc platform.opendatahub.io/instance.uid:191cdb15-01aa-4da4-9798-e6b508adbf9f platform.opendatahub.io/type:OpenShift AI Self-Managed platform.opendatahub.io/version:3.5.0-ea.2] creationTimestamp:2026-07-08T13:50:41Z finalizers:[platform.opendatahub.io/finalizer] generation:1 labels:map[platform.opendatahub.io/part-of:datasciencecluster] managedFields:[map[apiVersion:components.platform.opendatahub.io/v1alpha1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:component.opendatahub.io/management-state:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:finalizers:map[v:"platform.opendatahub.io/finalizer":map[]] f:labels:map[f:platform.opendatahub.io/part-of:map[]] f:ownerReferences:map[k:{"uid":"191cdb15-01aa-4da4-9798-e6b508adbf9f"}:map[]]] f:status:map[f:conditions:map[] f:observedGeneration:map[] f:phase:map[] f:releases:map[k:{"name":"KServe"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-inference-scheduler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-workload-variant-autoscaler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"vLLM"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]]] manager:kserve operation:Apply subresource:status time:2026-07-08T13:51:50Z] map[apiVersion:components.platform.opendatahub.io/v1alpha1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:component.opendatahub.io/management-state:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:labels:map[f:platform.opendatahub.io/part-of:map[]] f:ownerReferences:map[k:{"uid":"191cdb15-01aa-4da4-9798-e6b508adbf9f"}:map[]]] f:spec:map[f:modelsAsService:map[f:managementState:map[]] f:nim:map[] f:rawDeploymentServiceConfig:map[] f:wva:map[f:managementState:map[]]]] manager:datasciencecluster operation:Apply time:2026-07-08T13:56:08Z] map[apiVersion:components.platform.opendatahub.io/v1alpha1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:finalizers:map[.:map[] v:"platform.opendatahub.io/finalizer":map[]]]] manager:manager operation:Update time:2026-07-08T13:50:41Z]] name:default-kserve ownerReferences:[map[apiVersion:datasciencecluster.opendatahub.io/v2 blockOwnerDeletion:true controller:true kind:DataScienceCluster name:default-dsc uid:191cdb15-01aa-4da4-9798-e6b508adbf9f]] resourceVersion:384281026 uid:34fba53b-9c82-4975-b3cb-369ea3068e82] spec:map[modelsAsService:map[managementState:Managed] nim:map[airGapped:false managementState:Managed] rawDeploymentServiceConfig:Headless wva:map[managementState:Removed]] status:map[conditions:[map[lastTransitionTime:2026-07-08T13:51:49Z status:True type:Ready] map[lastTransitionTime:2026-07-08T13:51:49Z observedGeneration:1 status:True type:ProvisioningSucceeded] map[lastTransitionTime:2026-07-08T13:51:50Z observedGeneration:1 status:True type:DeploymentsAvailable] map[lastTransitionTime:2026-07-08T13:50:43Z observedGeneration:1 status:True type:DependenciesAvailable] map[lastTransitionTime:2026-07-08T13:50:43Z observedGeneration:1 status:True type:KserveLLMInferenceServiceDependencies] map[lastTransitionTime:2026-07-08T13:50:43Z observedGeneration:1 status:True type:KserveLLMInferenceServiceWideEPDependencies]] observedGeneration:1 phase:Ready releases:[map[name:KServe repoUrl:https://github.com/kserve/kserve/ version:v0.17.0] map[name:vLLM repoUrl:https://github.com/vllm-project/vllm version:v0.18.0] map[name:llm-d-inference-scheduler repoUrl:https://github.com/llm-d/llm-d-inference-scheduler version:v0.7.1] map[name:llm-d-workload-variant-autoscaler repoUrl:https://github.com/llm-d/llm-d-workload-variant-autoscaler version:v0.6.0]]]] to match expression <string>: .spec.nim.managementState == "" --- FAIL: TestOdhOperator/components/group_1/kserve (691.56s) === RUN TestOdhOperator/components/group_1 --- FAIL: TestOdhOperator/components/group_1 (0.00s) === RUN TestOdhOperator/components/group_2/kueue/Validate_autoCreateQueues_disabled_skips_queue_creation helper_test.go:156: Setting Kueue component (default-kueue) to Removed mode to start with a clean state. helper_test.go:156: Cleaning up any existing Kueue test resources. helper_test.go:156: Cleaning up Kueue cluster-scoped resources helper_test.go:156: Will attempt to delete 3 Kueue resources helper_test.go:156: Attempting to delete ClusterQueue /default helper_test.go:156: Successfully processed deletion of ClusterQueue /default helper_test.go:156: Attempting to delete ResourceFlavor /default-flavor helper_test.go:156: Successfully processed deletion of ResourceFlavor /default-flavor helper_test.go:156: Attempting to delete Kueue /cluster helper_test.go:156: Successfully processed deletion of Kueue /cluster helper_test.go:156: Ensuring LWS Operator is installed (namespace=openshift-lws-operator, name=leader-worker-set). helper_test.go:156: Ensuring LeaderWorkerSetOperator CR exists (namespace=openshift-lws-operator, name=cluster). helper_test.go:156: Ensuring OCP Kueue Operator is installed (namespace=openshift-kueue-operator, name=kueue-operator). helper_test.go:156: Creating test namespace (test-kueue-no-queues-d975h846tnlc7b09ahs0) with Kueue management annotation. helper_test.go:156: Creating Kueue ConfigMap (kueue-manager-config) to verify default Kueue resource is created correctly. helper_test.go:156: Creating Kueue ConfigMap (kueue-manager-config) in namespace (redhat-ods-applications). helper_test.go:156: Updating DSC to set Kueue to Unmanaged with autoCreateQueues=false (default). 2026-07-08T14:03:23Z INFO unknown field "spec.components.kueue.autoCreateQueues" helper_test.go:156: Verifying Kueue configuration (cluster) is still created. helper_test.go:156: Verifying no ClusterQueue, ResourceFlavor or LocalQueue is created (autoCreateQueues=false). testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_2/kueue/Validate_autoCreateQueues_disabled_skips_queue_creation (320.81s) === RUN TestOdhOperator/components/group_2/kueue === PAUSE TestOdhOperator/components/group_2/kueue === CONT TestOdhOperator/components/group_2/kueue 2026-07-08T14:02:00Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.19.18"}} test_context_test.go:1106: cleanup: best-effort delete failed for /, Kind= redhat-ods-applications/kueue-manager-config: Object 'Kind' is missing in 'unstructured object has no kind' test_context_test.go:615: Timed out after 300.009s. The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1824 with: Expected resource list 'default' of kind 'ClusterQueue' to be empty, but it contains resources. Expected <[]unstructured.Unstructured | len:1, cap:1>: [ { Object: { "apiVersion": <string>"kueue.x-k8s.io/v1beta2", "kind": <string>"ClusterQueue", "metadata": <map[string]interface {} | len:8>{ "creationTimestamp": <string>"2026-07-08T14:03:25Z", "generation": <int64>1, "labels": <map[string]interface {} | len:1>{ "platform.opendatahub.io/part-of": <string>"kueue", }, "managedFields": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:6>{ "manager": <string>"manager", "operation": <string>"Update", "time": <string>"2026-07-08T14:03:25Z", "apiVersion": <string>"kueue.x-k8s.io/v1beta2", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:2>{ "f:metadata": <map[string]interface {} | len:2>{ "f:annotations": <map[string]interface {} | len:6>{ "f:platform.opendatahub.io/instance.uid": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/type": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/version": <map[string]interface {} | len:0>{}, ".": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/instance.generation": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/instance.name": <map[string]interface {} | len:0>{}, }, "f:labels": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/part-of": <map[string]interface {} | len:0>{}, }, }, "f:spec": <map[string]interface {} | len:7>{ "f:namespaceSelector": <map[string]interface {} | len:0>{}, "f:preemption": <map[string]interface {} | len:4>{ ".": <map[string]interface {} | len:0>{}, "f:borrowWithinCohort": <map[string]interface {} | len:2>{...: ..., ...: ...}, "f:reclaimWithinCohort": <map[string]interface {} | len:0>{}, "f:withinClusterQueue": <map[string]interface {} | len:0>{}, }, "f:queueingStrategy": <map[string]interface {} | len:0>{}, "f:resourceGroups": <map[string]interface {} | len:0>{}, "f:stopPolicy": <map[string]interface {} | len:0>{}, ".": <map[string]interface {} | len:0>{}, "f:flavorFungibility": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:whenCanBorrow": <map[string]interface {} | len:0>{}, "f:whenCanPreempt": <map[string]interface {} | len:0>{}, }, }, }, }, ], "name": <string>"default", "resourceVersion": <string>"384288755", "uid": <string>"b947b6a9-dbd9-4e1c-9c4f-30e1258e7082", "annotations": <map[string]interface {} | len:5>{ "platform.opendatahub.io/instance.uid": <string>"a410ea5b-76af-4f30-8ead-c7d9af9f5c6f", "platform.opendatahub.io/type": <string>"OpenShift AI Self-Managed", "platform.opendatahub.io/version": <string>"3.5.0-ea.2", "platform.opendatahub.io/instance.generation": <string>"1", "platform.opendatahub.io/instance.name": <string>"default-kueue", }, }, "spec": <map[string]interface {} | len:6>{ "flavorFungibility": <map[string]interface {} | len:2>{ "whenCanBorrow": <string>"MayStopSearch", "whenCanPreempt": <string>"TryNextFlavor", }, "namespaceSelector": <map[string]interface {} | len:1>{ "matchLabels": <map[string]interface {} | len:1>{ "kueue.openshift.io/managed": <string>"true", }, }, "preemption": <map[string]interface {} | len:3>{ "reclaimWithinCohort": <string>"Never", "withinClusterQueue": <string>"Never", "borrowWithinCohort": <map[string]interface {} | len:1>{ "policy": <string>"Never", }, }, "queueingStrategy": <string>"BestEffortFIFO", "resourceGroups": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:2>{ "coveredResources": <[]interface {} | len:2, cap:2>[<string>"cpu", <string>"memory"], "flavors": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:2>{ "name": <string>"default-flavor", "resources": <[]interface {} | len:2, cap:2>[ <map[string]interface {} | len:2>{...: ..., ...: ...}, <map[string]interface {} | len:2>{...: ..., ...: ...}, ], }, ], }, ], "stopPolicy": <string>"None", }, }, }, ] to be empty test_context_test.go:1106: cleanup: best-effort delete failed for /, Kind= redhat-ods-applications/kueue-manager-config: Object 'Kind' is missing in 'unstructured object has no kind' --- FAIL: TestOdhOperator/components/group_2/kueue (393.08s) === RUN TestOdhOperator/components/group_2 --- FAIL: TestOdhOperator/components/group_2 (0.00s) === RUN TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable helper_test.go:156: Verifying Tenant models-as-a-service/default-tenant is present before disable helper_test.go:156: Disabling MaaS subcomponent (setting to Removed) helper_test.go:156: Waiting for Tenant models-as-a-service/default-tenant to be deleted testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable (430.15s) === RUN TestOdhOperator/components/group_4/modelsasservice === PAUSE TestOdhOperator/components/group_4/modelsasservice === CONT TestOdhOperator/components/group_4/modelsasservice 2026-07-08T14:12:13Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.19.18"}} controller_test.go:763: Creating MaaS PostgreSQL instance in namespace redhat-ods-applications controller_test.go:763: MaaS PostgreSQL instance and maas-db-config secret created in namespace redhat-ods-applications controller_test.go:763: Creating MaaS GatewayClass and Gateway: openshift-ingress/maas-default-gateway controller_test.go:763: Using hostname=maas.apps.ods-qe-psi-07.osp.rh-ods.com, certSecret=router-certs-default controller_test.go:763: MaaS Gateway openshift-ingress/maas-default-gateway created with HTTP+HTTPS listeners test_context_test.go:615: Timed out after 420.008s. The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1824 with: Expected resource list 'models-as-a-service/default-tenant' of kind 'Tenant' to be empty, but it contains resources. Expected <[]unstructured.Unstructured | len:1, cap:1>: [ { Object: { "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "kind": <string>"Tenant", "metadata": <map[string]interface {} | len:8>{ "name": <string>"default-tenant", "namespace": <string>"models-as-a-service", "ownerReferences": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:4>{ "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "kind": <string>"Config", "name": <string>"default", "uid": <string>"558d5405-f01f-4618-baf0-80fbc7b5ee68", }, ], "resourceVersion": <string>"384300180", "uid": <string>"01e7cd5e-61de-43f3-953b-14a53eefb4cf", "creationTimestamp": <string>"2026-07-08T14:13:02Z", "generation": <int64>1, "managedFields": <[]interface {} | len:2, cap:2>[ <map[string]interface {} | len:6>{ "fieldsV1": <map[string]interface {} | len:2>{ "f:metadata": <map[string]interface {} | len:1>{ "f:ownerReferences": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "k:{\"uid\":\"558d5405-f01f-4618-baf0-80fbc7b5ee68\"}": <map[string]interface {} | len:0>{}, }, }, "f:spec": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "f:gatewayRef": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:name": <map[string]interface {} | len:0>{}, "f:namespace": <map[string]interface {} | len:0>{}, }, }, }, "manager": <string>"manager", "operation": <string>"Update", "time": <string>"2026-07-08T14:13:02Z", "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "fieldsType": <string>"FieldsV1", }, <map[string]interface {} | len:7>{ "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:1>{ "f:status": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:conditions": <map[string]interface {} | len:0>{}, "f:phase": <map[string]interface {} | len:0>{}, }, }, "manager": <string>"manager", "operation": <string>"Update", "subresource": <string>"status", "time": <string>"2026-07-08T14:13:26Z", "apiVersion": <string>"maas.opendatahub.io/v1alpha1", }, ], }, "spec": <map[string]interface {} | len:1>{ "gatewayRef": <map[string]interface {} | len:2>{ "name": <string>"maas-default-gateway", "namespace": <string>"openshift-ingress", }, }, "status": <map[string]interface {} | len:2>{ "conditions": <[]interface {} | len:5, cap:8>[ <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-07-08T14:13:06Z", "message": <string>"AuthConfig CRD (Kuadrant) is available on the cluster", "observedGeneration": <int64>1, "reason": <string>"DependenciesMet", "status": <string>"True", "type": <string>"DependenciesAvailable", }, <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-07-08T14:13:06Z", "message": <string>"All prerequisites are satisfied", "observedGeneration": <int64>1, "reason": <string>"PrerequisitesMet", "status": <string>"True", "type": <string>"MaaSPrerequisitesAvailable", }, <map[string]interface {} | len:6>{ "status": <string>"False", "type": <string>"Degraded", "lastTransitionTime": <string>"2026-07-08T14:13:06Z", "message": <string>"All prerequisites are satisfied", "observedGeneration": <int64>1, "reason": <string>"PrerequisitesMet", }, <map[string]interface {} | len:6>{ "status": <string>"True", "type": <string>"DeploymentsAvailable", "lastTransitionTime": <string>"2026-07-08T14:13:26Z", "message": <string>"maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"DeploymentsReady", }, <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-07-08T14:13:26Z", "message": <string>"MaaS platform manifests applied and maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"Reconciled", "status": <string>"True", "type": <string>"Ready", }, ], "phase": <string>"Active", }, }, }, ] to be empty --- FAIL: TestOdhOperator/components/group_4/modelsasservice (512.58s) === RUN TestOdhOperator/components/group_4 --- FAIL: TestOdhOperator/components/group_4 (0.00s) === RUN TestOdhOperator/components 2026/07/08 14:01:59 TEST FAILURE DETECTED: group 1 2026/07/08 14:01:59 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 1) === 2026/07/08 14:02:00 === CLUSTER STATE === 2026/07/08 14:02:00 Node: ods-qe-psi-07-b4mqp-master-0 2026/07/08 14:02:00 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:02:00 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:02:00 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:02:00 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:02:00 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/08 14:02:00 Node: ods-qe-psi-07-b4mqp-master-1 2026/07/08 14:02:00 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:02:00 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:02:00 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:02:00 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:02:00 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/08 14:02:00 Node: ods-qe-psi-07-b4mqp-master-2 2026/07/08 14:02:00 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:02:00 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:02:00 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:02:00 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:02:00 Resources: 11500m CPU, 23450732Ki memory allocatable (of 12000m CPU, 24601708Ki memory capacity) 2026/07/08 14:02:00 Node: ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/08 14:02:00 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:02:00 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:02:00 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:02:00 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:02:00 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/08 14:02:00 Node: ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 14:02:00 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:02:00 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:02:00 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:02:00 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:02:00 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/08 14:02:00 Node: ods-qe-psi-07-b4mqp-worker-0-jd4pm 2026/07/08 14:02:00 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:02:00 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:02:00 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:02:00 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:02:00 Resources: 11500m CPU, 23452736Ki memory allocatable (of 12000m CPU, 24603712Ki memory capacity) 2026/07/08 14:02:00 Node: ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/08 14:02:00 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:02:00 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:02:00 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:02:00 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:02:00 Resources: 11500m CPU, 23452744Ki memory allocatable (of 12000m CPU, 24603720Ki memory capacity) 2026/07/08 14:02:00 Node: ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/08 14:02:00 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:02:00 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:02:00 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:02:00 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:02:00 Resources: 11500m CPU, 23450720Ki memory allocatable (of 12000m CPU, 24601696Ki memory capacity) 2026/07/08 14:02:00 === DEPLOYMENTS === 2026/07/08 14:02:00 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/08 14:02:00 === PODS === 2026/07/08 14:02:00 Failed to collect pod data: redhat-ods-applications/maas-controller-656b6867cb-8t9lr: container manager not ready; redhat-ods-applications/mlflow-668464596d-gtcrj: container combine-ca-bundles Completed (exit 0); redhat-ods-applications/trustyai-service-operator-controller-manager-6f7f88c874-9m8td: container manager restarts=2 2026/07/08 14:02:00 === OPERATOR STATUS === 2026/07/08 14:02:00 Operator deployment: 3/3 ready 2026/07/08 14:02:00 === DSCI STATUS === 2026/07/08 14:02:00 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/08 14:02:00 === DSC STATUS === 2026/07/08 14:02:00 DSC default-dsc: 2026/07/08 14:02:00 Ready: True - 2026/07/08 14:02:00 ProvisioningSucceeded: True - 2026/07/08 14:02:00 ComponentsReady: True - 2026/07/08 14:02:00 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/08 14:02:00 DashboardReady: False - Component ManagementState is set to Removed 2026/07/08 14:02:00 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/08 14:02:00 KserveLLMInferenceServiceDependencies: True - 2026/07/08 14:02:00 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/08 14:02:00 KserveReady: True - 2026/07/08 14:02:00 KueueReady: True - 2026/07/08 14:02:00 MLflowOperatorReady: True - 2026/07/08 14:02:00 MaaSPrerequisitesAvailable: True - MaaS gateway annotations and Authorino TLS are correctly configured 2026/07/08 14:02:00 ModelControllerReady: True - 2026/07/08 14:02:00 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/08 14:02:00 ModelsAsServiceReady: True - MaaS platform manifests applied and maas-api deployment is available 2026/07/08 14:02:00 OGXReady: False - Component ManagementState is set to Removed 2026/07/08 14:02:00 RayReady: False - Component ManagementState is set to Removed 2026/07/08 14:02:00 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/08 14:02:00 TrainerReady: False - Component ManagementState is set to Removed 2026/07/08 14:02:00 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/08 14:02:00 TrustyAIReady: True - 2026/07/08 14:02:00 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/08 14:02:00 === RECENT EVENTS === 2026/07/08 14:02:00 14:01:46 INFO Pod/maas-controller-656b6867cb-8t9lr: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:b8cca0d56d9ca5ab366cdc4042f13a908326e57b1b711c59e6ff101507058fb4" in 1.047s (1.047s including waiting). Image size: 212345923 bytes. 2026/07/08 14:02:00 14:01:46 INFO Pod/maas-controller-656b6867cb-8t9lr: Created - Created container: manager 2026/07/08 14:02:00 14:01:45 INFO Pod/maas-controller-656b6867cb-8t9lr: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:b8cca0d56d9ca5ab366cdc4042f13a908326e57b1b711c59e6ff101507058fb4" 2026/07/08 14:02:00 14:01:19 INFO Pod/mlflow-668464596d-gtcrj: Created - Created container: ca-bundle-watcher 2026/07/08 14:02:00 14:01:19 INFO Pod/mlflow-668464596d-gtcrj: Started - Started container ca-bundle-watcher 2026/07/08 14:02:00 14:01:19 INFO Pod/mlflow-668464596d-gtcrj: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mlflow-rhel9@sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e" in 1.782s (1.782s including waiting). Image size: 1216832503 bytes. 2026/07/08 14:02:00 14:01:17 INFO Pod/mlflow-668464596d-gtcrj: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mlflow-rhel9@sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e" in 1.641s (1.641s including waiting). Image size: 1216832503 bytes. 2026/07/08 14:02:00 14:01:17 INFO Pod/mlflow-668464596d-gtcrj: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mlflow-rhel9@sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e" 2026/07/08 14:02:00 14:01:17 INFO Pod/mlflow-668464596d-gtcrj: Started - Started container mlflow 2026/07/08 14:02:00 14:01:17 INFO Pod/mlflow-668464596d-gtcrj: Created - Created container: mlflow 2026/07/08 14:02:00 14:01:16 INFO Pod/mlflow-668464596d-gtcrj: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mlflow-rhel9@sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e" 2026/07/08 14:02:00 14:01:15 INFO Pod/mlflow-668464596d-gtcrj: Started - Started container combine-ca-bundles 2026/07/08 14:02:00 14:01:15 INFO Pod/mlflow-668464596d-gtcrj: Created - Created container: combine-ca-bundles 2026/07/08 14:02:00 14:01:14 INFO Pod/mlflow-668464596d-gtcrj: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mlflow-rhel9@sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e" 2026/07/08 14:02:00 14:01:14 INFO Pod/mlflow-668464596d-gtcrj: AddedInterface - Add eth0 [10.129.2.228/23] from ovn-kubernetes 2026/07/08 14:02:00 14:01:14 INFO Pod/mlflow-668464596d-gtcrj: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mlflow-rhel9@sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e" in 763ms (763ms including waiting). Image size: 1216832503 bytes. 2026/07/08 14:02:00 14:01:06 INFO Pod/mlflow-668464596d-gtcrj: SuccessfulAttachVolume - AttachVolume.Attach succeeded for volume "pvc-9b7de459-4cc6-482d-b261-ff0de611028a" 2026/07/08 14:02:00 14:01:03 INFO Pod/mlflow-668464596d-gtcrj: Scheduled - Successfully assigned redhat-ods-applications/mlflow-668464596d-gtcrj to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 14:02:00 14:01:03 INFO Deployment/mlflow: ScalingReplicaSet - Scaled up replica set mlflow-668464596d from 0 to 1 2026/07/08 14:02:00 14:01:03 INFO Job/mlflow-mg-3120-g1: Completed - Job completed 2026/07/08 14:02:00 14:01:03 INFO ReplicaSet/mlflow-668464596d: SuccessfulCreate - Created pod: mlflow-668464596d-gtcrj 2026/07/08 14:02:00 14:00:56 INFO Pod/mlflow-mg-3120-g1-c96wb: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mlflow-rhel9@sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e" 2026/07/08 14:02:00 14:00:56 INFO Pod/mlflow-mg-3120-g1-c96wb: Started - Started container db-migrate 2026/07/08 14:02:00 14:00:56 INFO Pod/mlflow-mg-3120-g1-c96wb: Created - Created container: db-migrate 2026/07/08 14:02:00 14:00:56 INFO Pod/mlflow-mg-3120-g1-c96wb: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mlflow-rhel9@sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e" in 719ms (719ms including waiting). Image size: 1216832503 bytes. 2026/07/08 14:02:00 14:00:54 INFO Pod/mlflow-mg-3120-g1-c96wb: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mlflow-rhel9@sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e" in 948ms (948ms including waiting). Image size: 1216832503 bytes. 2026/07/08 14:02:00 14:00:54 INFO Pod/mlflow-mg-3120-g1-c96wb: Started - Started container combine-ca-bundles 2026/07/08 14:02:00 14:00:54 INFO Pod/mlflow-mg-3120-g1-c96wb: Created - Created container: combine-ca-bundles 2026/07/08 14:02:00 14:00:53 INFO Pod/mlflow-mg-3120-g1-c96wb: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mlflow-rhel9@sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e" 2026/07/08 14:02:00 14:00:53 INFO Pod/mlflow-mg-3120-g1-c96wb: AddedInterface - Add eth0 [10.129.2.227/23] from ovn-kubernetes 2026/07/08 14:02:00 14:00:52 WARN Pod/maas-controller-656b6867cb-8t9lr: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-khkwx); attempting to pull the image may not succeed. 2026/07/08 14:02:00 14:00:47 INFO Pod/mlflow-mg-3120-g1-c96wb: SuccessfulAttachVolume - AttachVolume.Attach succeeded for volume "pvc-9b7de459-4cc6-482d-b261-ff0de611028a" 2026/07/08 14:02:00 14:00:45 INFO Pod/mlflow-mg-3120-g1-c96wb: Scheduled - Successfully assigned redhat-ods-applications/mlflow-mg-3120-g1-c96wb to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 14:02:00 14:00:44 INFO Job/mlflow-mg-3120-g1: SuccessfulCreate - Created pod: mlflow-mg-3120-g1-c96wb 2026/07/08 14:02:00 14:00:44 INFO PersistentVolumeClaim/mlflow-pvc: Provisioning - External provisioner is provisioning volume for claim "redhat-ods-applications/mlflow-pvc" 2026/07/08 14:02:00 14:00:44 INFO PersistentVolumeClaim/mlflow-pvc: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'cinder.csi.openstack.org' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. 2026/07/08 14:02:00 14:00:44 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/08 14:02:00 14:00:44 INFO PersistentVolumeClaim/mlflow-pvc: WaitForFirstConsumer - waiting for first consumer to be created before binding 2026/07/08 14:02:00 14:00:44 INFO PersistentVolumeClaim/mlflow-pvc: ProvisioningSucceeded - Successfully provisioned volume pvc-9b7de459-4cc6-482d-b261-ff0de611028a 2026/07/08 14:02:00 14:00:42 INFO Pod/postgres-deployment-649dd85bb5-lrdpb: AddedInterface - Add eth0 [10.130.2.153/23] from ovn-kubernetes 2026/07/08 14:02:00 14:00:42 INFO Pod/postgres-deployment-649dd85bb5-lrdpb: Started - Started container postgres 2026/07/08 14:02:00 14:00:42 INFO Pod/postgres-deployment-649dd85bb5-lrdpb: Created - Created container: postgres 2026/07/08 14:02:00 14:00:42 INFO Pod/postgres-deployment-649dd85bb5-lrdpb: Pulled - Container image "registry.redhat.io/rhel9/postgresql-15@sha256:90ec347a35ab8a5d530c8d09f5347b13cc71df04f3b994bfa8b1a409b1171d59" already present on machine 2026/07/08 14:02:00 14:00:41 INFO Pod/postgres-deployment-649dd85bb5-lrdpb: SuccessfulAttachVolume - AttachVolume.Attach succeeded for volume "pvc-b4861810-0f57-4de9-9da3-f337c5b53537" 2026/07/08 14:02:00 14:00:27 INFO Pod/postgres-deployment-649dd85bb5-lrdpb: Scheduled - Successfully assigned redhat-ods-applications/postgres-deployment-649dd85bb5-lrdpb to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/08 14:02:00 14:00:27 WARN Pod/postgres-deployment-649dd85bb5-lrdpb: FailedAttachVolume - Multi-Attach error for volume "pvc-b4861810-0f57-4de9-9da3-f337c5b53537" Volume is already exclusively attached to one node and can't be attached to another 2026/07/08 14:02:00 14:00:27 INFO Deployment/postgres-deployment: ScalingReplicaSet - Scaled up replica set postgres-deployment-649dd85bb5 from 0 to 1 2026/07/08 14:02:00 14:00:27 INFO ReplicaSet/postgres-deployment-649dd85bb5: SuccessfulCreate - Created pod: postgres-deployment-649dd85bb5-lrdpb 2026/07/08 14:02:00 14:00:26 INFO Pod/postgres-deployment-869f977f94-4vlxq: Killing - Stopping container postgres 2026/07/08 14:02:00 14:00:24 INFO Deployment/postgres-deployment: ScalingReplicaSet - Scaled down replica set postgres-deployment-869f977f94 from 1 to 0 2026/07/08 14:02:00 14:00:24 INFO ReplicaSet/postgres-deployment-869f977f94: SuccessfulDelete - Deleted pod: postgres-deployment-869f977f94-4vlxq 2026/07/08 14:02:00 14:00:23 INFO Pod/postgres-deployment-869f977f94-4vlxq: Started - Started container postgres 2026/07/08 14:02:00 14:00:23 INFO Pod/postgres-deployment-869f977f94-4vlxq: Created - Created container: postgres 2026/07/08 14:02:00 14:00:23 INFO Pod/postgres-deployment-869f977f94-4vlxq: Pulled - Successfully pulled image "registry.redhat.io/rhel9/postgresql-15@sha256:90ec347a35ab8a5d530c8d09f5347b13cc71df04f3b994bfa8b1a409b1171d59" in 7.439s (7.439s including waiting). Image size: 404709194 bytes. 2026/07/08 14:02:00 14:00:16 INFO Pod/postgres-deployment-869f977f94-4vlxq: Pulling - Pulling image "registry.redhat.io/rhel9/postgresql-15@sha256:90ec347a35ab8a5d530c8d09f5347b13cc71df04f3b994bfa8b1a409b1171d59" 2026/07/08 14:02:00 14:00:16 INFO Pod/postgres-deployment-869f977f94-4vlxq: AddedInterface - Add eth0 [10.131.1.228/23] from ovn-kubernetes 2026/07/08 14:02:00 14:00:14 INFO Pod/postgres-deployment-869f977f94-4vlxq: SuccessfulAttachVolume - AttachVolume.Attach succeeded for volume "pvc-b4861810-0f57-4de9-9da3-f337c5b53537" 2026/07/08 14:02:00 14:00:12 INFO Pod/postgres-deployment-869f977f94-4vlxq: Scheduled - Successfully assigned redhat-ods-applications/postgres-deployment-869f977f94-4vlxq to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/08 14:02:00 14:00:11 INFO ReplicaSet/postgres-deployment-869f977f94: SuccessfulCreate - Created pod: postgres-deployment-869f977f94-4vlxq 2026/07/08 14:02:00 14:00:11 INFO PersistentVolumeClaim/postgres-pvc: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'cinder.csi.openstack.org' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. 2026/07/08 14:02:00 14:00:11 INFO PersistentVolumeClaim/postgres-pvc: Provisioning - External provisioner is provisioning volume for claim "redhat-ods-applications/postgres-pvc" 2026/07/08 14:02:00 14:00:11 INFO PersistentVolumeClaim/postgres-pvc: ProvisioningSucceeded - Successfully provisioned volume pvc-b4861810-0f57-4de9-9da3-f337c5b53537 2026/07/08 14:02:00 14:00:10 INFO Deployment/postgres-deployment: ScalingReplicaSet - Scaled up replica set postgres-deployment-869f977f94 from 0 to 1 2026/07/08 14:02:00 14:00:10 INFO PersistentVolumeClaim/postgres-pvc: WaitForFirstConsumer - waiting for first consumer to be created before binding 2026/07/08 14:02:00 14:00:04 INFO CronJob/maas-api-key-cleanup: SawCompletedJob - Saw completed job: maas-api-key-cleanup-29725320, condition: Complete 2026/07/08 14:02:00 14:00:04 INFO Job/maas-api-key-cleanup-29725320: Completed - Job completed 2026/07/08 14:02:00 14:00:04 INFO CronJob/maas-api-key-cleanup: SuccessfulDelete - Deleted job maas-api-key-cleanup-29725275 2026/07/08 14:02:00 14:00:01 INFO Pod/maas-api-key-cleanup-29725320-xsvql: Created - Created container: cleanup 2026/07/08 14:02:00 14:00:01 INFO Pod/maas-api-key-cleanup-29725320-xsvql: Started - Started container cleanup 2026/07/08 14:02:00 14:00:00 INFO Pod/maas-api-key-cleanup-29725320-xsvql: Scheduled - Successfully assigned redhat-ods-applications/maas-api-key-cleanup-29725320-xsvql to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/08 14:02:00 14:00:00 INFO Job/maas-api-key-cleanup-29725320: SuccessfulCreate - Created pod: maas-api-key-cleanup-29725320-xsvql 2026/07/08 14:02:00 14:00:00 INFO Pod/maas-api-key-cleanup-29725320-xsvql: Pulled - Container image "registry.redhat.io/ubi9/ubi-minimal@sha256:907b68736aa798b2d38255b7aa070b2a70acb90803864a40f05d0ec47556ddd0" already present on machine 2026/07/08 14:02:00 14:00:00 INFO Pod/maas-api-key-cleanup-29725320-xsvql: AddedInterface - Add eth0 [10.131.1.227/23] from ovn-kubernetes 2026/07/08 14:02:00 14:00:00 INFO CronJob/maas-api-key-cleanup: SuccessfulCreate - Created job maas-api-key-cleanup-29725320 2026/07/08 14:02:00 13:57:58 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - ServiceMonitor controller-manager-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/08 14:02:00 === RESOURCE QUOTAS === 2026/07/08 14:02:00 No resource quotas found in monitored namespaces 2026/07/08 14:02:00 Classification for group 1: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/08 14:02:00 Evidence: container maas-controller-656b6867cb-8t9lr/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-656b6867cb-8t9lr_redhat-ods-applications(09191a10-a6c7-4aa4-bf3d-db73992acc4b) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-656b6867cb-8t9lr/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-656b6867cb-8t9lr_redhat-ods-applications(09191a10-a6c7-4aa4-bf3d-db73992acc4b)"],"confidence":"medium"} 2026/07/08 14:02:00 Diagnostics complete for failed test: group 1 controller_test.go:763: Stopping: group 1 test failed. 2026/07/08 14:02:00 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/08 14:02:00 Circuit breaker: infrastructure/unknown failure recorded (2/3) 2026/07/08 14:08:33 TEST FAILURE DETECTED: group 2 2026/07/08 14:08:33 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 2) === 2026/07/08 14:08:34 === CLUSTER STATE === 2026/07/08 14:08:34 Node: ods-qe-psi-07-b4mqp-master-0 2026/07/08 14:08:34 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:08:34 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:08:34 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:08:34 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:08:34 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/08 14:08:34 Node: ods-qe-psi-07-b4mqp-master-1 2026/07/08 14:08:34 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:08:34 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:08:34 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:08:34 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:08:34 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/08 14:08:34 Node: ods-qe-psi-07-b4mqp-master-2 2026/07/08 14:08:34 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:08:34 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:08:34 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:08:34 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:08:34 Resources: 11500m CPU, 23450732Ki memory allocatable (of 12000m CPU, 24601708Ki memory capacity) 2026/07/08 14:08:34 Node: ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/08 14:08:34 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:08:34 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:08:34 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:08:34 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:08:34 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/08 14:08:34 Node: ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 14:08:34 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:08:34 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:08:34 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:08:34 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:08:34 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/08 14:08:34 Node: ods-qe-psi-07-b4mqp-worker-0-jd4pm 2026/07/08 14:08:34 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:08:34 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:08:34 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:08:34 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:08:34 Resources: 11500m CPU, 23452736Ki memory allocatable (of 12000m CPU, 24603712Ki memory capacity) 2026/07/08 14:08:34 Node: ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/08 14:08:34 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:08:34 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:08:34 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:08:34 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:08:34 Resources: 11500m CPU, 23452744Ki memory allocatable (of 12000m CPU, 24603720Ki memory capacity) 2026/07/08 14:08:34 Node: ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/08 14:08:34 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:08:34 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:08:34 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:08:34 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:08:34 Resources: 11500m CPU, 23450720Ki memory allocatable (of 12000m CPU, 24601696Ki memory capacity) 2026/07/08 14:08:34 === DEPLOYMENTS === 2026/07/08 14:08:34 Namespace: redhat-ods-applications 2026/07/08 14:08:34 Deployment maas-postgres: 1/1 (READY) 2026/07/08 14:08:34 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/08 14:08:34 Deployment postgres: 1/1 (READY) 2026/07/08 14:08:34 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/08 14:08:34 Namespace: kube-system 2026/07/08 14:08:34 === PODS === 2026/07/08 14:08:34 Failed to collect pod data: redhat-ods-applications/trustyai-service-operator-controller-manager-6f7f88c874-9m8td: container manager restarts=2 2026/07/08 14:08:34 === OPERATOR STATUS === 2026/07/08 14:08:34 Operator deployment: 3/3 ready 2026/07/08 14:08:34 === DSCI STATUS === 2026/07/08 14:08:34 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/08 14:08:34 === DSC STATUS === 2026/07/08 14:08:34 Failed to collect DSC data: default-dsc unhealthy conditions: ModelsAsServiceReady=False 2026/07/08 14:08:34 === RECENT EVENTS === 2026/07/08 14:08:34 14:06:52 WARN Pod/mlflow-58f77fb56c-zf7nl: Unhealthy - Readiness probe failed: Get "https://10.129.2.233:8443/mlflow/health": dial tcp 10.129.2.233:8443: connect: connection refused 2026/07/08 14:08:34 14:06:28 INFO Pod/postgres-deployment-6fbcc5c6c8-fb22v: Killing - Stopping container postgres 2026/07/08 14:08:34 14:06:27 INFO Pod/mlflow-58f77fb56c-zf7nl: Killing - Stopping container ca-bundle-watcher 2026/07/08 14:08:34 14:06:27 INFO Pod/mlflow-58f77fb56c-zf7nl: Killing - Stopping container mlflow 2026/07/08 14:08:34 14:05:15 INFO Pod/workload-variant-autoscaler-controller-manager-56865c9848-ncbgf: Killing - Stopping container manager 2026/07/08 14:08:34 14:05:15 INFO Pod/model-serving-api-78bbbb9bf8-nz8v4: Killing - Stopping container server 2026/07/08 14:08:34 14:05:15 INFO Pod/model-registry-operator-controller-manager-5f68f4fd76-njpnd: Killing - Stopping container manager 2026/07/08 14:08:34 14:05:15 INFO Pod/kserve-controller-manager-6c8cb86d7f-pgd27: Killing - Stopping container manager 2026/07/08 14:08:34 14:05:15 INFO Pod/llmisvc-controller-manager-586c85c465-wj2dr: Killing - Stopping container manager 2026/07/08 14:08:34 14:05:15 INFO Pod/maas-api-5f6cf7d65c-m4jtc: Killing - Stopping container maas-api 2026/07/08 14:08:34 14:05:15 INFO Pod/odh-model-controller-6b647f87c-qvb6f: Killing - Stopping container manager 2026/07/08 14:08:34 14:05:00 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6b647f87c-qvb6f_30d2cfc0-6cd7-4f91-9b53-9f2d0adeefe0 became leader 2026/07/08 14:08:34 14:04:59 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-56865c9848-ncbgf_f135c0f2-a66a-40a7-b144-d431ed7abb02 became leader 2026/07/08 14:08:34 14:04:58 INFO Pod/workload-variant-autoscaler-controller-manager-56865c9848-ncbgf: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-56865c9848-ncbgf to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/08 14:08:34 14:04:58 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-56865c9848 from 0 to 1 2026/07/08 14:08:34 14:04:58 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-56865c9848: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-56865c9848-ncbgf 2026/07/08 14:08:34 14:04:58 INFO Pod/workload-variant-autoscaler-controller-manager-56865c9848-hmlr7: Killing - Stopping container manager 2026/07/08 14:08:34 14:04:58 INFO Pod/workload-variant-autoscaler-controller-manager-56865c9848-ncbgf: AddedInterface - Add eth0 [10.130.2.160/23] from ovn-kubernetes 2026/07/08 14:08:34 14:04:58 INFO Pod/workload-variant-autoscaler-controller-manager-56865c9848-nqs97: Killing - Stopping container manager 2026/07/08 14:08:34 14:04:58 INFO Pod/workload-variant-autoscaler-controller-manager-56865c9848-ncbgf: Created - Created container: manager 2026/07/08 14:08:34 14:04:58 INFO Pod/workload-variant-autoscaler-controller-manager-56865c9848-ncbgf: Pulled - Container image "registry.redhat.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:3d219851cf91eb92db3df70276a7e0615e56f41deeff9a7b83b5e93b686bb789" already present on machine 2026/07/08 14:08:34 14:04:58 INFO Pod/workload-variant-autoscaler-controller-manager-56865c9848-ncbgf: Started - Started container manager 2026/07/08 14:08:34 14:04:44 INFO Pod/odh-model-controller-6b647f87c-qvb6f: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b423702816000ce0f075548e3a67de006f40ccf05c575a866cb62e2ac9711191" in 908ms (908ms including waiting). Image size: 238719956 bytes. 2026/07/08 14:08:34 14:04:44 INFO Pod/odh-model-controller-6b647f87c-qvb6f: Started - Started container manager 2026/07/08 14:08:34 14:04:44 INFO Pod/odh-model-controller-6b647f87c-qvb6f: Created - Created container: manager 2026/07/08 14:08:34 14:04:43 INFO Pod/odh-model-controller-6b647f87c-qvb6f: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6b647f87c-qvb6f to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/08 14:08:34 14:04:43 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6b647f87c from 0 to 1 2026/07/08 14:08:34 14:04:43 INFO Pod/odh-model-controller-6b647f87c-qvb6f: AddedInterface - Add eth0 [10.130.2.159/23] from ovn-kubernetes 2026/07/08 14:08:34 14:04:43 INFO ReplicaSet/odh-model-controller-6b647f87c: SuccessfulCreate - Created pod: odh-model-controller-6b647f87c-qvb6f 2026/07/08 14:08:34 14:04:43 INFO Pod/odh-model-controller-6b647f87c-qvb6f: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b423702816000ce0f075548e3a67de006f40ccf05c575a866cb62e2ac9711191" 2026/07/08 14:08:34 14:04:42 INFO Pod/odh-model-controller-6b647f87c-mklhd: Killing - Stopping container manager 2026/07/08 14:08:34 14:04:42 INFO Pod/odh-model-controller-6b647f87c-9zmtt: Killing - Stopping container manager 2026/07/08 14:08:34 14:04:28 INFO Pod/model-serving-api-78bbbb9bf8-nz8v4: Created - Created container: server 2026/07/08 14:08:34 14:04:28 INFO Pod/model-serving-api-78bbbb9bf8-nz8v4: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:6dd699e678c3f4d6e51138eea62f096f26472b5f9c1ce2757cf9e9e2f80371a4" already present on machine 2026/07/08 14:08:34 14:04:28 INFO Pod/model-serving-api-78bbbb9bf8-nz8v4: Started - Started container server 2026/07/08 14:08:34 14:04:28 INFO Pod/model-serving-api-78bbbb9bf8-nz8v4: AddedInterface - Add eth0 [10.130.2.157/23] from ovn-kubernetes 2026/07/08 14:08:34 14:04:27 INFO Pod/model-serving-api-78bbbb9bf8-nz8v4: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-78bbbb9bf8-nz8v4 to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/08 14:08:34 14:04:27 INFO ReplicaSet/model-serving-api-78bbbb9bf8: SuccessfulCreate - Created pod: model-serving-api-78bbbb9bf8-nz8v4 2026/07/08 14:08:34 14:04:27 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-78bbbb9bf8 from 0 to 1 2026/07/08 14:08:34 14:04:27 INFO Pod/model-serving-api-78bbbb9bf8-h6qj5: Killing - Stopping container server 2026/07/08 14:08:34 14:04:27 INFO Pod/model-serving-api-78bbbb9bf8-nhv5v: Killing - Stopping container server 2026/07/08 14:08:34 14:04:25 INFO Pod/mlflow-58f77fb56c-zf7nl: Created - Created container: ca-bundle-watcher 2026/07/08 14:08:34 14:04:25 INFO Pod/mlflow-58f77fb56c-zf7nl: Started - Started container ca-bundle-watcher 2026/07/08 14:08:34 14:04:25 INFO Pod/mlflow-58f77fb56c-zf7nl: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mlflow-rhel9@sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e" in 1.314s (1.314s including waiting). Image size: 1216832503 bytes. 2026/07/08 14:08:34 14:04:24 INFO Pod/mlflow-58f77fb56c-zf7nl: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mlflow-rhel9@sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e" in 1.234s (1.234s including waiting). Image size: 1216832503 bytes. 2026/07/08 14:08:34 14:04:24 INFO Pod/mlflow-58f77fb56c-zf7nl: Created - Created container: mlflow 2026/07/08 14:08:34 14:04:24 INFO Pod/mlflow-58f77fb56c-zf7nl: Started - Started container mlflow 2026/07/08 14:08:34 14:04:24 INFO Pod/mlflow-58f77fb56c-zf7nl: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mlflow-rhel9@sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e" 2026/07/08 14:08:34 14:04:22 INFO Pod/mlflow-58f77fb56c-zf7nl: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mlflow-rhel9@sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e" 2026/07/08 14:08:34 14:04:21 INFO Pod/mlflow-58f77fb56c-zf7nl: Started - Started container combine-ca-bundles 2026/07/08 14:08:34 14:04:21 INFO Pod/mlflow-58f77fb56c-zf7nl: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mlflow-rhel9@sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e" in 1.013s (1.013s including waiting). Image size: 1216832503 bytes. 2026/07/08 14:08:34 14:04:21 INFO Pod/mlflow-58f77fb56c-zf7nl: Created - Created container: combine-ca-bundles 2026/07/08 14:08:34 14:04:20 INFO Pod/mlflow-58f77fb56c-zf7nl: Scheduled - Successfully assigned redhat-ods-applications/mlflow-58f77fb56c-zf7nl to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 14:08:34 14:04:20 INFO ReplicaSet/mlflow-58f77fb56c: SuccessfulCreate - Created pod: mlflow-58f77fb56c-zf7nl 2026/07/08 14:08:34 14:04:20 INFO Pod/mlflow-58f77fb56c-zf7nl: AddedInterface - Add eth0 [10.129.2.233/23] from ovn-kubernetes 2026/07/08 14:08:34 14:04:20 INFO Pod/mlflow-58f77fb56c-zf7nl: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mlflow-rhel9@sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e" 2026/07/08 14:08:34 14:04:20 INFO Deployment/mlflow: ScalingReplicaSet - Scaled up replica set mlflow-58f77fb56c from 0 to 1 2026/07/08 14:08:34 14:04:19 INFO Job/mlflow-mg-3120-g1: Completed - Job completed 2026/07/08 14:08:34 14:04:13 INFO Pod/mlflow-mg-3120-g1-5sjrb: Started - Started container db-migrate 2026/07/08 14:08:34 14:04:13 INFO Pod/mlflow-mg-3120-g1-5sjrb: Created - Created container: db-migrate 2026/07/08 14:08:34 14:04:13 INFO Pod/mlflow-mg-3120-g1-5sjrb: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mlflow-rhel9@sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e" in 847ms (847ms including waiting). Image size: 1216832503 bytes. 2026/07/08 14:08:34 14:04:12 INFO Pod/mlflow-mg-3120-g1-5sjrb: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mlflow-rhel9@sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e" 2026/07/08 14:08:34 14:04:11 INFO Pod/mlflow-mg-3120-g1-5sjrb: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mlflow-rhel9@sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e" in 928ms (928ms including waiting). Image size: 1216832503 bytes. 2026/07/08 14:08:34 14:04:11 INFO Pod/mlflow-mg-3120-g1-5sjrb: Started - Started container combine-ca-bundles 2026/07/08 14:08:34 14:04:11 INFO Pod/mlflow-mg-3120-g1-5sjrb: Created - Created container: combine-ca-bundles 2026/07/08 14:08:34 14:04:10 INFO Pod/mlflow-mg-3120-g1-5sjrb: Scheduled - Successfully assigned redhat-ods-applications/mlflow-mg-3120-g1-5sjrb to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 14:08:34 14:04:10 INFO Pod/mlflow-mg-3120-g1-5sjrb: AddedInterface - Add eth0 [10.129.2.232/23] from ovn-kubernetes 2026/07/08 14:08:34 14:04:10 INFO Pod/mlflow-mg-3120-g1-5sjrb: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mlflow-rhel9@sha256:df8acfff1e5f9bdf1d98d32da6fafcb1cdfe539201ca287c5ebd772da653b49e" 2026/07/08 14:08:34 14:04:10 INFO Job/mlflow-mg-3120-g1: SuccessfulCreate - Created pod: mlflow-mg-3120-g1-5sjrb 2026/07/08 14:08:34 14:04:07 INFO Pod/postgres-deployment-6fbcc5c6c8-fb22v: Started - Started container postgres 2026/07/08 14:08:34 14:04:07 INFO Pod/postgres-deployment-6fbcc5c6c8-fb22v: Pulled - Container image "registry.redhat.io/rhel9/postgresql-15@sha256:90ec347a35ab8a5d530c8d09f5347b13cc71df04f3b994bfa8b1a409b1171d59" already present on machine 2026/07/08 14:08:34 14:04:07 INFO Pod/postgres-deployment-6fbcc5c6c8-fb22v: AddedInterface - Add eth0 [10.129.2.230/23] from ovn-kubernetes 2026/07/08 14:08:34 14:04:07 INFO Pod/postgres-deployment-6fbcc5c6c8-fb22v: Created - Created container: postgres 2026/07/08 14:08:34 14:04:00 INFO Pod/postgres-deployment-6fbcc5c6c8-fb22v: Scheduled - Successfully assigned redhat-ods-applications/postgres-deployment-6fbcc5c6c8-fb22v to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 14:08:34 14:04:00 INFO Deployment/postgres-deployment: ScalingReplicaSet - Scaled up replica set postgres-deployment-6fbcc5c6c8 from 0 to 1 2026/07/08 14:08:34 14:04:00 INFO ReplicaSet/postgres-deployment-6fbcc5c6c8: SuccessfulCreate - Created pod: postgres-deployment-6fbcc5c6c8-fb22v 2026/07/08 14:08:34 14:03:59 INFO Pod/postgres-deployment-869f977f94-zq4t9: Killing - Stopping container postgres 2026/07/08 14:08:34 14:03:58 INFO ReplicaSet/postgres-deployment-869f977f94: SuccessfulDelete - Deleted pod: postgres-deployment-869f977f94-zq4t9 2026/07/08 14:08:34 14:03:58 INFO Deployment/postgres-deployment: ScalingReplicaSet - Scaled down replica set postgres-deployment-869f977f94 from 1 to 0 2026/07/08 14:08:34 14:03:57 INFO Pod/postgres-deployment-869f977f94-zq4t9: Started - Started container postgres 2026/07/08 14:08:34 14:03:57 INFO Pod/postgres-deployment-869f977f94-zq4t9: Created - Created container: postgres 2026/07/08 14:08:34 14:03:57 INFO Pod/postgres-deployment-869f977f94-zq4t9: Pulled - Container image "registry.redhat.io/rhel9/postgresql-15@sha256:90ec347a35ab8a5d530c8d09f5347b13cc71df04f3b994bfa8b1a409b1171d59" already present on machine 2026/07/08 14:08:34 14:03:57 INFO Pod/postgres-deployment-869f977f94-zq4t9: AddedInterface - Add eth0 [10.129.2.229/23] from ovn-kubernetes 2026/07/08 14:08:34 14:03:52 INFO Pod/postgres-deployment-869f977f94-zq4t9: SuccessfulAttachVolume - AttachVolume.Attach succeeded for volume "pvc-c299931a-7db1-4dac-8ca3-31c1198d2bb5" 2026/07/08 14:08:34 14:03:50 INFO Pod/postgres-deployment-869f977f94-zq4t9: Scheduled - Successfully assigned redhat-ods-applications/postgres-deployment-869f977f94-zq4t9 to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 14:08:34 14:03:49 INFO ReplicaSet/postgres-deployment-869f977f94: SuccessfulCreate - Created pod: postgres-deployment-869f977f94-zq4t9 2026/07/08 14:08:34 14:03:49 INFO PersistentVolumeClaim/postgres-pvc: ProvisioningSucceeded - Successfully provisioned volume pvc-c299931a-7db1-4dac-8ca3-31c1198d2bb5 2026/07/08 14:08:34 14:03:49 INFO PersistentVolumeClaim/postgres-pvc: Provisioning - External provisioner is provisioning volume for claim "redhat-ods-applications/postgres-pvc" 2026/07/08 14:08:34 14:03:49 INFO PersistentVolumeClaim/postgres-pvc: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'cinder.csi.openstack.org' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. 2026/07/08 14:08:34 14:03:49 INFO PersistentVolumeClaim/postgres-pvc: WaitForFirstConsumer - waiting for first consumer to be created before binding 2026/07/08 14:08:34 14:03:49 INFO Deployment/postgres-deployment: ScalingReplicaSet - Scaled up replica set postgres-deployment-869f977f94 from 0 to 1 2026/07/08 14:08:34 14:03:41 WARN Pod/mlflow-668464596d-gtcrj: Unhealthy - Readiness probe failed: Get "https://10.129.2.228:8443/mlflow/health": dial tcp 10.129.2.228:8443: connect: connection refused 2026/07/08 14:08:34 === RESOURCE QUOTAS === 2026/07/08 14:08:34 No resource quotas found in monitored namespaces 2026/07/08 14:08:34 Classification for group 2: infrastructure/probe-failure (code=1014, confidence=medium) 2026/07/08 14:08:34 Evidence: event Pod/mlflow-58f77fb56c-zf7nl: Unhealthy - Readiness probe failed: Get "https://10.129.2.233:8443/mlflow/health": dial tcp 10.129.2.233:8443: connect: connection refused FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"probe-failure","error_code":1014,"evidence":["event Pod/mlflow-58f77fb56c-zf7nl: Unhealthy - Readiness probe failed: Get \"https://10.129.2.233:8443/mlflow/health\": dial tcp 10.129.2.233:8443: connect: connection refused"],"confidence":"medium"} 2026/07/08 14:08:34 Diagnostics complete for failed test: group 2 controller_test.go:763: Stopping: group 2 test failed. 2026/07/08 14:08:34 Circuit breaker: failure classified as infrastructure/probe-failure (code=1014) 2026/07/08 14:08:34 Circuit breaker: infrastructure/unknown failure recorded (3/3) 2026/07/08 14:08:34 Circuit breaker: threshold reached (3 failures), running confirming health check... 2026/07/08 14:08:34 Circuit breaker: confirming health check passed — cluster appears healthy, resetting counter 2026/07/08 14:20:45 TEST FAILURE DETECTED: group 4 2026/07/08 14:20:45 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 4) === 2026/07/08 14:20:46 === CLUSTER STATE === 2026/07/08 14:20:46 Node: ods-qe-psi-07-b4mqp-master-0 2026/07/08 14:20:46 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:20:46 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:20:46 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:20:46 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:20:46 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/08 14:20:46 Node: ods-qe-psi-07-b4mqp-master-1 2026/07/08 14:20:46 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:20:46 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:20:46 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:20:46 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:20:46 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/08 14:20:46 Node: ods-qe-psi-07-b4mqp-master-2 2026/07/08 14:20:46 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:20:46 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:20:46 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:20:46 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:20:46 Resources: 11500m CPU, 23450732Ki memory allocatable (of 12000m CPU, 24601708Ki memory capacity) 2026/07/08 14:20:46 Node: ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/08 14:20:46 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:20:46 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:20:46 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:20:46 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:20:46 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/08 14:20:46 Node: ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 14:20:46 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:20:46 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:20:46 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:20:46 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:20:46 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/08 14:20:46 Node: ods-qe-psi-07-b4mqp-worker-0-jd4pm 2026/07/08 14:20:46 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:20:46 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:20:46 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:20:46 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:20:46 Resources: 11500m CPU, 23452736Ki memory allocatable (of 12000m CPU, 24603712Ki memory capacity) 2026/07/08 14:20:46 Node: ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/08 14:20:46 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:20:46 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:20:46 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:20:46 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:20:46 Resources: 11500m CPU, 23452744Ki memory allocatable (of 12000m CPU, 24603720Ki memory capacity) 2026/07/08 14:20:46 Node: ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/08 14:20:46 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:20:46 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:20:46 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:20:46 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:20:46 Resources: 11500m CPU, 23450720Ki memory allocatable (of 12000m CPU, 24601696Ki memory capacity) 2026/07/08 14:20:46 === DEPLOYMENTS === 2026/07/08 14:20:46 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/08 14:20:46 === PODS === 2026/07/08 14:20:46 Failed to collect pod data: redhat-ods-applications/maas-controller-656b6867cb-x6k8q: container manager not ready 2026/07/08 14:20:46 === OPERATOR STATUS === 2026/07/08 14:20:46 Operator deployment: 3/3 ready 2026/07/08 14:20:46 === DSCI STATUS === 2026/07/08 14:20:46 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/08 14:20:46 === DSC STATUS === 2026/07/08 14:20:46 DSC default-dsc: 2026/07/08 14:20:46 Ready: True - 2026/07/08 14:20:46 ProvisioningSucceeded: True - 2026/07/08 14:20:46 ComponentsReady: True - 2026/07/08 14:20:46 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/08 14:20:46 DashboardReady: False - Component ManagementState is set to Removed 2026/07/08 14:20:46 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/08 14:20:46 KserveLLMInferenceServiceDependencies: True - 2026/07/08 14:20:46 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/08 14:20:46 KserveReady: True - 2026/07/08 14:20:46 KueueReady: True - 2026/07/08 14:20:46 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/08 14:20:46 ModelControllerReady: True - 2026/07/08 14:20:46 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/08 14:20:46 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/08 14:20:46 OGXReady: False - Component ManagementState is set to Removed 2026/07/08 14:20:46 RayReady: False - Component ManagementState is set to Removed 2026/07/08 14:20:46 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/08 14:20:46 TrainerReady: False - Component ManagementState is set to Removed 2026/07/08 14:20:46 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/08 14:20:46 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/08 14:20:46 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/08 14:20:46 === RECENT EVENTS === 2026/07/08 14:20:46 14:19:53 INFO Pod/maas-controller-656b6867cb-x6k8q: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:b8cca0d56d9ca5ab366cdc4042f13a908326e57b1b711c59e6ff101507058fb4" in 1.237s (1.237s including waiting). Image size: 212345923 bytes. 2026/07/08 14:20:46 14:19:52 INFO Pod/maas-controller-656b6867cb-x6k8q: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:b8cca0d56d9ca5ab366cdc4042f13a908326e57b1b711c59e6ff101507058fb4" 2026/07/08 14:20:46 14:18:36 WARN Pod/maas-controller-656b6867cb-x6k8q: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-qsmdx); attempting to pull the image may not succeed. 2026/07/08 14:20:46 14:17:01 INFO Pod/maas-controller-656b6867cb-x6k8q: Created - Created container: manager 2026/07/08 14:20:46 14:17:01 INFO Pod/maas-controller-656b6867cb-x6k8q: Started - Started container manager 2026/07/08 14:20:46 14:17:01 INFO Pod/maas-controller-656b6867cb-x6k8q: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:b8cca0d56d9ca5ab366cdc4042f13a908326e57b1b711c59e6ff101507058fb4" in 736ms (736ms including waiting). Image size: 212345923 bytes. 2026/07/08 14:20:46 === RESOURCE QUOTAS === 2026/07/08 14:20:46 No resource quotas found in monitored namespaces 2026/07/08 14:20:46 Classification for group 4: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/08 14:20:46 Evidence: container maas-controller-656b6867cb-x6k8q/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-656b6867cb-x6k8q_redhat-ods-applications(490e9749-1ff1-4762-9c2d-30c18194de6f) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-656b6867cb-x6k8q/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-656b6867cb-x6k8q_redhat-ods-applications(490e9749-1ff1-4762-9c2d-30c18194de6f)"],"confidence":"medium"} 2026/07/08 14:20:46 Diagnostics complete for failed test: group 4 controller_test.go:763: Stopping: group 4 test failed. 2026/07/08 14:20:46 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/08 14:20:46 Circuit breaker: infrastructure/unknown failure recorded (1/3) --- FAIL: TestOdhOperator/components (1961.62s) === RUN TestOdhOperator 2026/07/08 13:45:16 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/08 13:50:27 TEST FAILURE DETECTED: DSCInitialization and DataScienceCluster validation E2E Tests 2026/07/08 13:50:27 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:DSCInitialization and DataScienceCluster validation E2E Tests) === 2026/07/08 13:50:28 === CLUSTER STATE === 2026/07/08 13:50:28 Node: ods-qe-psi-07-b4mqp-master-0 2026/07/08 13:50:28 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 13:50:28 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 13:50:28 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 13:50:28 Condition Ready: True - kubelet is posting ready status 2026/07/08 13:50:28 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/08 13:50:28 Node: ods-qe-psi-07-b4mqp-master-1 2026/07/08 13:50:28 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 13:50:28 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 13:50:28 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 13:50:28 Condition Ready: True - kubelet is posting ready status 2026/07/08 13:50:28 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/08 13:50:28 Node: ods-qe-psi-07-b4mqp-master-2 2026/07/08 13:50:28 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 13:50:28 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 13:50:28 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 13:50:28 Condition Ready: True - kubelet is posting ready status 2026/07/08 13:50:28 Resources: 11500m CPU, 23450732Ki memory allocatable (of 12000m CPU, 24601708Ki memory capacity) 2026/07/08 13:50:28 Node: ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/08 13:50:28 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 13:50:28 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 13:50:28 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 13:50:28 Condition Ready: True - kubelet is posting ready status 2026/07/08 13:50:28 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/08 13:50:28 Node: ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 13:50:28 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 13:50:28 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 13:50:28 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 13:50:28 Condition Ready: True - kubelet is posting ready status 2026/07/08 13:50:28 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/08 13:50:28 Node: ods-qe-psi-07-b4mqp-worker-0-jd4pm 2026/07/08 13:50:28 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 13:50:28 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 13:50:28 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 13:50:28 Condition Ready: True - kubelet is posting ready status 2026/07/08 13:50:28 Resources: 11500m CPU, 23452736Ki memory allocatable (of 12000m CPU, 24603712Ki memory capacity) 2026/07/08 13:50:28 Node: ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/08 13:50:28 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 13:50:28 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 13:50:28 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 13:50:28 Condition Ready: True - kubelet is posting ready status 2026/07/08 13:50:28 Resources: 11500m CPU, 23452744Ki memory allocatable (of 12000m CPU, 24603720Ki memory capacity) 2026/07/08 13:50:28 Node: ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/08 13:50:28 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 13:50:28 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 13:50:28 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 13:50:28 Condition Ready: True - kubelet is posting ready status 2026/07/08 13:50:28 Resources: 11500m CPU, 23450720Ki memory allocatable (of 12000m CPU, 24601696Ki memory capacity) 2026/07/08 13:50:28 === DEPLOYMENTS === 2026/07/08 13:50:28 Namespace: redhat-ods-applications 2026/07/08 13:50:28 Deployment dashboard-redirect: 2/2 (READY) 2026/07/08 13:50:28 Deployment data-science-pipelines-operator-controller-manager: 1/1 (READY) 2026/07/08 13:50:28 Deployment feast-operator-controller-manager: 1/1 (READY) 2026/07/08 13:50:28 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/08 13:50:28 Deployment kubeflow-trainer-controller-manager: 1/1 (READY) 2026/07/08 13:50:28 Deployment kubeflow-training-operator: 1/1 (READY) 2026/07/08 13:50:28 Deployment kuberay-operator: 1/1 (READY) 2026/07/08 13:50:28 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/08 13:50:28 Deployment maas-api: 1/1 (READY) 2026/07/08 13:50:28 Deployment maas-controller: 1/1 (READY) 2026/07/08 13:50:28 Deployment maas-postgres: 1/1 (READY) 2026/07/08 13:50:28 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/08 13:50:28 Deployment model-registry-operator-controller-manager: 1/1 (READY) 2026/07/08 13:50:28 Deployment model-serving-api: 1/1 (READY) 2026/07/08 13:50:28 Deployment notebook-controller-deployment: 1/1 (READY) 2026/07/08 13:50:28 Deployment odh-model-controller: 1/1 (READY) 2026/07/08 13:50:28 Deployment odh-notebook-controller-manager: 1/1 (READY) 2026/07/08 13:50:28 Deployment ogx-k8s-operator-controller-manager: 1/1 (READY) 2026/07/08 13:50:28 Deployment postgres: 1/1 (READY) 2026/07/08 13:50:28 Deployment rhods-dashboard: 2/2 (READY) 2026/07/08 13:50:28 Deployment spark-operator-controller: 1/1 (READY) 2026/07/08 13:50:28 Deployment spark-operator-webhook: 1/1 (READY) 2026/07/08 13:50:28 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/08 13:50:28 Namespace: kube-system 2026/07/08 13:50:28 === PODS === 2026/07/08 13:50:28 Failed to collect pod data: redhat-ods-applications/trustyai-service-operator-controller-manager-6f7f88c874-9m8td: container manager restarts=2 2026/07/08 13:50:28 === OPERATOR STATUS === 2026/07/08 13:50:28 Operator deployment: 3/3 ready 2026/07/08 13:50:28 === DSCI STATUS === 2026/07/08 13:50:28 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/08 13:50:28 === DSC STATUS === 2026/07/08 13:50:28 DSC default-dsc: 2026/07/08 13:50:28 Ready: True - 2026/07/08 13:50:28 ProvisioningSucceeded: True - 2026/07/08 13:50:28 ModulesReady: True - 2026/07/08 13:50:28 AIPipelinesReady: True - 2026/07/08 13:50:28 ComponentsReady: True - 2026/07/08 13:50:28 DashboardReady: True - 2026/07/08 13:50:28 FeastOperatorReady: True - 2026/07/08 13:50:28 KserveLLMInferenceServiceDependencies: True - 2026/07/08 13:50:28 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/08 13:50:28 KserveReady: True - 2026/07/08 13:50:28 KueueReady: True - 2026/07/08 13:50:28 MLflowOperatorReady: True - 2026/07/08 13:50:28 MaaSPrerequisitesAvailable: True - MaaS gateway annotations and Authorino TLS are correctly configured 2026/07/08 13:50:28 ModelControllerReady: True - 2026/07/08 13:50:28 ModelRegistryReady: True - 2026/07/08 13:50:28 ModelsAsServiceReady: True - MaaS platform manifests applied and maas-api deployment is available 2026/07/08 13:50:28 OGXReady: True - 2026/07/08 13:50:28 RayReady: True - 2026/07/08 13:50:28 SparkOperatorReady: True - 2026/07/08 13:50:28 TrainerReady: True - 2026/07/08 13:50:28 TrainingOperatorReady: True - 2026/07/08 13:50:28 TrustyAIReady: True - 2026/07/08 13:50:28 WorkbenchesReady: True - 2026/07/08 13:50:28 === RECENT EVENTS === 2026/07/08 13:50:28 No recent events found in monitored namespaces 2026/07/08 13:50:28 === RESOURCE QUOTAS === 2026/07/08 13:50:28 No resource quotas found in monitored namespaces 2026/07/08 13:50:28 Classification for DSCInitialization and DataScienceCluster validation E2E Tests: infrastructure/dsci-unhealthy (code=1009, confidence=medium) 2026/07/08 13:50:28 Evidence: DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"dsci-unhealthy","error_code":1009,"evidence":["DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False"],"confidence":"medium"} 2026/07/08 13:50:28 Diagnostics complete for failed test: DSCInitialization and DataScienceCluster validation E2E Tests controller_test.go:478: Stopping: DSCInitialization and DataScienceCluster validation E2E Tests test failed. 2026/07/08 13:50:28 Circuit breaker: failure classified as infrastructure/dsci-unhealthy (code=1009) 2026/07/08 13:50:28 Circuit breaker: infrastructure/unknown failure recorded (1/3) 2026/07/08 14:23:09 TEST FAILURE DETECTED: components 2026/07/08 14:23:09 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/08 14:23:10 === CLUSTER STATE === 2026/07/08 14:23:10 Node: ods-qe-psi-07-b4mqp-master-0 2026/07/08 14:23:10 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:23:10 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:23:10 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:23:10 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:23:10 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/08 14:23:10 Node: ods-qe-psi-07-b4mqp-master-1 2026/07/08 14:23:10 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:23:10 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:23:10 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:23:10 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:23:10 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/08 14:23:10 Node: ods-qe-psi-07-b4mqp-master-2 2026/07/08 14:23:10 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:23:10 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:23:10 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:23:10 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:23:10 Resources: 11500m CPU, 23450732Ki memory allocatable (of 12000m CPU, 24601708Ki memory capacity) 2026/07/08 14:23:10 Node: ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/08 14:23:10 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:23:10 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:23:10 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:23:10 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:23:10 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/08 14:23:10 Node: ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 14:23:10 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:23:10 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:23:10 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:23:10 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:23:10 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/08 14:23:10 Node: ods-qe-psi-07-b4mqp-worker-0-jd4pm 2026/07/08 14:23:10 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:23:10 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:23:10 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:23:10 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:23:10 Resources: 11500m CPU, 23452736Ki memory allocatable (of 12000m CPU, 24603712Ki memory capacity) 2026/07/08 14:23:10 Node: ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/08 14:23:10 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:23:10 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:23:10 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:23:10 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:23:10 Resources: 11500m CPU, 23452744Ki memory allocatable (of 12000m CPU, 24603720Ki memory capacity) 2026/07/08 14:23:10 Node: ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/08 14:23:10 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 14:23:10 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 14:23:10 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 14:23:10 Condition Ready: True - kubelet is posting ready status 2026/07/08 14:23:10 Resources: 11500m CPU, 23450720Ki memory allocatable (of 12000m CPU, 24601696Ki memory capacity) 2026/07/08 14:23:10 === DEPLOYMENTS === 2026/07/08 14:23:10 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/08 14:23:10 === PODS === 2026/07/08 14:23:10 Failed to collect pod data: redhat-ods-applications/maas-controller-656b6867cb-x6k8q: container manager not ready 2026/07/08 14:23:10 === OPERATOR STATUS === 2026/07/08 14:23:10 Operator deployment: 3/3 ready 2026/07/08 14:23:10 === DSCI STATUS === 2026/07/08 14:23:10 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/08 14:23:10 === DSC STATUS === 2026/07/08 14:23:10 DSC default-dsc: 2026/07/08 14:23:10 Ready: True - 2026/07/08 14:23:10 ProvisioningSucceeded: True - 2026/07/08 14:23:10 ComponentsReady: True - 2026/07/08 14:23:10 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/08 14:23:10 DashboardReady: False - Component ManagementState is set to Removed 2026/07/08 14:23:10 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/08 14:23:10 KserveLLMInferenceServiceDependencies: True - 2026/07/08 14:23:10 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/08 14:23:10 KserveReady: True - 2026/07/08 14:23:10 KueueReady: True - 2026/07/08 14:23:10 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/08 14:23:10 ModelControllerReady: True - 2026/07/08 14:23:10 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/08 14:23:10 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/08 14:23:10 OGXReady: False - Component ManagementState is set to Removed 2026/07/08 14:23:10 RayReady: False - Component ManagementState is set to Removed 2026/07/08 14:23:10 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/08 14:23:10 TrainerReady: True - 2026/07/08 14:23:10 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/08 14:23:10 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/08 14:23:10 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/08 14:23:10 === RECENT EVENTS === 2026/07/08 14:23:10 14:22:44 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-586c85c465-c6hbr_b285e735-01c3-45f5-b8bc-2af4f15d32ef became leader 2026/07/08 14:23:10 14:22:43 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c8cb86d7f-jcx8n_12cb97da-b1fe-4917-a01c-80f6f203723c became leader 2026/07/08 14:23:10 14:22:28 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-54bf759f64-8lpjk_851adefb-d5c5-449b-b374-ea5812652367 became leader 2026/07/08 14:23:10 14:22:26 INFO Pod/kserve-controller-manager-6c8cb86d7f-jcx8n: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:7f6f393635a7431d530256ef8ad48ebd35531166dbb361c6c0ff9fc37bd5dc2b" in 1.22s (1.22s including waiting). Image size: 237680379 bytes. 2026/07/08 14:23:10 14:22:26 INFO Pod/kserve-controller-manager-6c8cb86d7f-jcx8n: Created - Created container: manager 2026/07/08 14:23:10 14:22:26 INFO Pod/kserve-controller-manager-6c8cb86d7f-jcx8n: Started - Started container manager 2026/07/08 14:23:10 14:22:25 INFO Pod/llmisvc-controller-manager-586c85c465-c6hbr: Started - Started container manager 2026/07/08 14:23:10 14:22:25 INFO Pod/llmisvc-controller-manager-586c85c465-c6hbr: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:d3225eb3f0bfcb20a6d1dccd63403e58a810921680425ecae1b22d86a451417d" in 889ms (889ms including waiting). Image size: 235185383 bytes. 2026/07/08 14:23:10 14:22:25 INFO Pod/llmisvc-controller-manager-586c85c465-c6hbr: Created - Created container: manager 2026/07/08 14:23:10 14:22:24 INFO Pod/llmisvc-controller-manager-586c85c465-c6hbr: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-586c85c465-c6hbr to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 14:23:10 14:22:24 INFO Pod/kserve-controller-manager-6c8cb86d7f-jcx8n: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c8cb86d7f-jcx8n to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/08 14:23:10 14:22:24 INFO Pod/llmisvc-controller-manager-586c85c465-c6hbr: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:d3225eb3f0bfcb20a6d1dccd63403e58a810921680425ecae1b22d86a451417d" 2026/07/08 14:23:10 14:22:24 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-586c85c465 from 0 to 1 2026/07/08 14:23:10 14:22:24 INFO Pod/llmisvc-controller-manager-586c85c465-c6hbr: AddedInterface - Add eth0 [10.129.2.253/23] from ovn-kubernetes 2026/07/08 14:23:10 14:22:24 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c8cb86d7f from 0 to 1 2026/07/08 14:23:10 14:22:24 INFO ReplicaSet/kserve-controller-manager-6c8cb86d7f: SuccessfulCreate - Created pod: kserve-controller-manager-6c8cb86d7f-jcx8n 2026/07/08 14:23:10 14:22:24 INFO Pod/kserve-controller-manager-6c8cb86d7f-jcx8n: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:7f6f393635a7431d530256ef8ad48ebd35531166dbb361c6c0ff9fc37bd5dc2b" 2026/07/08 14:23:10 14:22:24 INFO Pod/kserve-controller-manager-6c8cb86d7f-jcx8n: AddedInterface - Add eth0 [10.131.1.241/23] from ovn-kubernetes 2026/07/08 14:23:10 14:22:24 INFO ReplicaSet/llmisvc-controller-manager-586c85c465: SuccessfulCreate - Created pod: llmisvc-controller-manager-586c85c465-c6hbr 2026/07/08 14:23:10 14:22:12 INFO Pod/odh-model-controller-54bf759f64-8lpjk: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b423702816000ce0f075548e3a67de006f40ccf05c575a866cb62e2ac9711191" in 938ms (938ms including waiting). Image size: 238719956 bytes. 2026/07/08 14:23:10 14:22:12 INFO Pod/model-serving-api-78bbbb9bf8-llb4h: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:6dd699e678c3f4d6e51138eea62f096f26472b5f9c1ce2757cf9e9e2f80371a4" already present on machine 2026/07/08 14:23:10 14:22:12 INFO Pod/model-serving-api-78bbbb9bf8-llb4h: AddedInterface - Add eth0 [10.129.2.252/23] from ovn-kubernetes 2026/07/08 14:23:10 14:22:12 INFO Pod/odh-model-controller-54bf759f64-8lpjk: Started - Started container manager 2026/07/08 14:23:10 14:22:12 INFO Pod/odh-model-controller-54bf759f64-8lpjk: Created - Created container: manager 2026/07/08 14:23:10 14:22:12 INFO Pod/model-serving-api-78bbbb9bf8-llb4h: Created - Created container: server 2026/07/08 14:23:10 14:22:12 INFO Pod/model-serving-api-78bbbb9bf8-llb4h: Started - Started container server 2026/07/08 14:23:10 14:22:11 INFO Pod/model-serving-api-78bbbb9bf8-llb4h: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-78bbbb9bf8-llb4h to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 14:23:10 14:22:11 INFO Pod/odh-model-controller-54bf759f64-8lpjk: AddedInterface - Add eth0 [10.131.1.240/23] from ovn-kubernetes 2026/07/08 14:23:10 14:22:11 WARN ReplicaSet/model-serving-api-78bbbb9bf8: FailedCreate - Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/08 14:23:10 14:22:11 INFO ReplicaSet/model-serving-api-78bbbb9bf8: SuccessfulCreate - Created pod: model-serving-api-78bbbb9bf8-llb4h 2026/07/08 14:23:10 14:22:11 INFO Pod/odh-model-controller-54bf759f64-8lpjk: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b423702816000ce0f075548e3a67de006f40ccf05c575a866cb62e2ac9711191" 2026/07/08 14:23:10 14:22:11 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/08 14:23:10 14:22:11 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-78bbbb9bf8 from 0 to 1 2026/07/08 14:23:10 14:22:10 INFO Pod/odh-model-controller-54bf759f64-8lpjk: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-54bf759f64-8lpjk to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/08 14:23:10 14:22:10 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-54bf759f64 from 0 to 1 2026/07/08 14:23:10 14:22:10 WARN Pod/odh-model-controller-54bf759f64-8lpjk: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/08 14:23:10 14:22:10 INFO ReplicaSet/odh-model-controller-54bf759f64: SuccessfulCreate - Created pod: odh-model-controller-54bf759f64-8lpjk 2026/07/08 14:23:10 14:21:59 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/08 14:23:10 14:21:59 INFO Pod/model-serving-api-78bbbb9bf8-t7r7z: Killing - Stopping container server 2026/07/08 14:23:10 14:21:59 INFO Pod/odh-model-controller-54bf759f64-b9gj5: Killing - Stopping container manager 2026/07/08 14:23:10 14:21:59 INFO Pod/kserve-controller-manager-6c8cb86d7f-kfztr: Killing - Stopping container manager 2026/07/08 14:23:10 14:21:59 INFO Pod/llmisvc-controller-manager-586c85c465-jxgqw: Killing - Stopping container manager 2026/07/08 14:23:10 14:21:35 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-586c85c465-jxgqw_885314a8-316a-4cdf-8e63-ec50e608f5d2 became leader 2026/07/08 14:23:10 14:21:32 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c8cb86d7f-kfztr_445bc6e7-d188-4597-9c69-aea22bd071ea became leader 2026/07/08 14:23:10 14:21:18 INFO Pod/llmisvc-controller-manager-586c85c465-jxgqw: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:d3225eb3f0bfcb20a6d1dccd63403e58a810921680425ecae1b22d86a451417d" in 4.578s (4.578s including waiting). Image size: 235185383 bytes. 2026/07/08 14:23:10 14:21:18 INFO Pod/llmisvc-controller-manager-586c85c465-jxgqw: Started - Started container manager 2026/07/08 14:23:10 14:21:18 INFO Pod/llmisvc-controller-manager-586c85c465-jxgqw: Created - Created container: manager 2026/07/08 14:23:10 14:21:16 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-54bf759f64-b9gj5_cedfd0d8-778f-446c-96f3-335f0f946422 became leader 2026/07/08 14:23:10 14:21:14 INFO Pod/kserve-controller-manager-6c8cb86d7f-kfztr: Started - Started container manager 2026/07/08 14:23:10 14:21:14 INFO Pod/kserve-controller-manager-6c8cb86d7f-kfztr: Created - Created container: manager 2026/07/08 14:23:10 14:21:14 INFO Pod/kserve-controller-manager-6c8cb86d7f-kfztr: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:7f6f393635a7431d530256ef8ad48ebd35531166dbb361c6c0ff9fc37bd5dc2b" in 1.172s (1.172s including waiting). Image size: 237680379 bytes. 2026/07/08 14:23:10 14:21:13 INFO Pod/llmisvc-controller-manager-586c85c465-jxgqw: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-586c85c465-jxgqw to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 14:23:10 14:21:13 INFO Pod/kserve-controller-manager-6c8cb86d7f-kfztr: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c8cb86d7f-kfztr to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/08 14:23:10 14:21:13 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-586c85c465 from 0 to 1 2026/07/08 14:23:10 14:21:13 INFO Pod/kserve-controller-manager-6c8cb86d7f-kfztr: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:7f6f393635a7431d530256ef8ad48ebd35531166dbb361c6c0ff9fc37bd5dc2b" 2026/07/08 14:23:10 14:21:13 INFO Pod/llmisvc-controller-manager-586c85c465-jxgqw: AddedInterface - Add eth0 [10.129.2.251/23] from ovn-kubernetes 2026/07/08 14:23:10 14:21:13 INFO Pod/llmisvc-controller-manager-586c85c465-jxgqw: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:d3225eb3f0bfcb20a6d1dccd63403e58a810921680425ecae1b22d86a451417d" 2026/07/08 14:23:10 14:21:13 INFO ReplicaSet/llmisvc-controller-manager-586c85c465: SuccessfulCreate - Created pod: llmisvc-controller-manager-586c85c465-jxgqw 2026/07/08 14:23:10 14:21:13 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c8cb86d7f from 0 to 1 2026/07/08 14:23:10 14:21:13 INFO ReplicaSet/kserve-controller-manager-6c8cb86d7f: SuccessfulCreate - Created pod: kserve-controller-manager-6c8cb86d7f-kfztr 2026/07/08 14:23:10 14:21:13 INFO Pod/kserve-controller-manager-6c8cb86d7f-kfztr: AddedInterface - Add eth0 [10.131.1.239/23] from ovn-kubernetes 2026/07/08 14:23:10 14:21:08 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-776d978f97-7ms2g_5b056a44-7187-4cd2-9971-77d530a1dda4 became leader 2026/07/08 14:23:10 14:21:01 INFO Pod/model-serving-api-78bbbb9bf8-t7r7z: AddedInterface - Add eth0 [10.130.2.164/23] from ovn-kubernetes 2026/07/08 14:23:10 14:21:01 INFO Pod/model-serving-api-78bbbb9bf8-t7r7z: Started - Started container server 2026/07/08 14:23:10 14:21:01 INFO Pod/model-serving-api-78bbbb9bf8-t7r7z: Created - Created container: server 2026/07/08 14:23:10 14:21:01 INFO Pod/model-serving-api-78bbbb9bf8-t7r7z: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:6dd699e678c3f4d6e51138eea62f096f26472b5f9c1ce2757cf9e9e2f80371a4" already present on machine 2026/07/08 14:23:10 14:21:00 INFO Pod/model-serving-api-78bbbb9bf8-t7r7z: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-78bbbb9bf8-t7r7z to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/08 14:23:10 14:21:00 INFO Pod/odh-model-controller-54bf759f64-b9gj5: Started - Started container manager 2026/07/08 14:23:10 14:21:00 INFO ReplicaSet/model-serving-api-78bbbb9bf8: SuccessfulCreate - Created pod: model-serving-api-78bbbb9bf8-t7r7z 2026/07/08 14:23:10 14:21:00 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-78bbbb9bf8 from 0 to 1 2026/07/08 14:23:10 14:21:00 WARN ReplicaSet/model-serving-api-78bbbb9bf8: FailedCreate - Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/08 14:23:10 14:21:00 INFO Pod/odh-model-controller-54bf759f64-b9gj5: Created - Created container: manager 2026/07/08 14:23:10 14:21:00 INFO Pod/odh-model-controller-54bf759f64-b9gj5: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b423702816000ce0f075548e3a67de006f40ccf05c575a866cb62e2ac9711191" in 829ms (829ms including waiting). Image size: 238719956 bytes. 2026/07/08 14:23:10 14:20:59 INFO Pod/odh-model-controller-54bf759f64-b9gj5: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-54bf759f64-b9gj5 to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 14:23:10 14:20:59 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-54bf759f64 from 0 to 1 2026/07/08 14:23:10 14:20:59 INFO Pod/odh-model-controller-54bf759f64-b9gj5: AddedInterface - Add eth0 [10.129.2.250/23] from ovn-kubernetes 2026/07/08 14:23:10 14:20:59 INFO Pod/odh-model-controller-54bf759f64-b9gj5: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b423702816000ce0f075548e3a67de006f40ccf05c575a866cb62e2ac9711191" 2026/07/08 14:23:10 14:20:59 INFO ReplicaSet/odh-model-controller-54bf759f64: SuccessfulCreate - Created pod: odh-model-controller-54bf759f64-b9gj5 2026/07/08 14:23:10 14:20:52 INFO Pod/kubeflow-trainer-controller-manager-776d978f97-7ms2g: Created - Created container: manager 2026/07/08 14:23:10 14:20:52 INFO Pod/kubeflow-trainer-controller-manager-776d978f97-7ms2g: Pulled - Container image "registry.redhat.io/rhoai/odh-trainer-rhel9@sha256:904dd86da976fced2d3618cbe932baadaf94934280aab3a68f6cc9f04ee2efbd" already present on machine 2026/07/08 14:23:10 14:20:52 INFO Pod/kubeflow-trainer-controller-manager-776d978f97-7ms2g: Started - Started container manager 2026/07/08 14:23:10 14:20:52 INFO Pod/kubeflow-trainer-controller-manager-776d978f97-7ms2g: AddedInterface - Add eth0 [10.129.2.249/23] from ovn-kubernetes 2026/07/08 14:23:10 14:20:51 INFO Pod/kubeflow-trainer-controller-manager-776d978f97-7ms2g: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-776d978f97-7ms2g to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 14:23:10 14:20:51 INFO ReplicaSet/kubeflow-trainer-controller-manager-776d978f97: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-776d978f97-7ms2g 2026/07/08 14:23:10 14:20:51 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-776d978f97 from 0 to 1 2026/07/08 14:23:10 14:20:48 INFO Pod/model-serving-api-78bbbb9bf8-bq8rr: Killing - Stopping container server 2026/07/08 14:23:10 14:20:48 INFO Pod/odh-model-controller-54bf759f64-5qc6d: Killing - Stopping container manager 2026/07/08 14:23:10 14:20:47 INFO Pod/kserve-controller-manager-6c8cb86d7f-txw44: Killing - Stopping container manager 2026/07/08 14:23:10 14:20:47 INFO Pod/llmisvc-controller-manager-586c85c465-87zr5: Killing - Stopping container manager 2026/07/08 14:23:10 14:19:53 INFO Pod/maas-controller-656b6867cb-x6k8q: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:b8cca0d56d9ca5ab366cdc4042f13a908326e57b1b711c59e6ff101507058fb4" in 1.237s (1.237s including waiting). Image size: 212345923 bytes. 2026/07/08 14:23:10 14:19:52 INFO Pod/maas-controller-656b6867cb-x6k8q: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:b8cca0d56d9ca5ab366cdc4042f13a908326e57b1b711c59e6ff101507058fb4" 2026/07/08 14:23:10 14:18:36 WARN Pod/maas-controller-656b6867cb-x6k8q: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-qsmdx); attempting to pull the image may not succeed. 2026/07/08 14:23:10 === RESOURCE QUOTAS === 2026/07/08 14:23:10 No resource quotas found in monitored namespaces 2026/07/08 14:23:10 Classification for components: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/08 14:23:10 Evidence: container maas-controller-656b6867cb-x6k8q/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-656b6867cb-x6k8q_redhat-ods-applications(490e9749-1ff1-4762-9c2d-30c18194de6f) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-656b6867cb-x6k8q/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-656b6867cb-x6k8q_redhat-ods-applications(490e9749-1ff1-4762-9c2d-30c18194de6f)"],"confidence":"medium"} 2026/07/08 14:23:10 Diagnostics complete for failed test: components controller_test.go:491: Stopping: components test failed. 2026/07/08 14:23:10 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/08 14:23:10 Circuit breaker: infrastructure/unknown failure recorded (2/3) Deletion Policy: Never. Skipping deletion tests. 2026/07/08 14:23:12 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (2275.96s)