=== RUN TestOdhOperator/DSCInitialization_and_DataScienceCluster_validation_E2E_Tests/Validate_DSCI_webhook_denies_invalid_PEM_on_update testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/DSCInitialization_and_DataScienceCluster_validation_E2E_Tests/Validate_DSCI_webhook_denies_invalid_PEM_on_update (300.01s) === RUN TestOdhOperator/DSCInitialization_and_DataScienceCluster_validation_E2E_Tests 2026-07-08T18:56:09Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.19.18"}} test_context_test.go:1675: Timed out after 300.008s. Webhook should deny update with invalid PEM in customCABundle The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1664 with: Webhook should deny update with invalid PEM in customCABundle Expected an error to have occurred. Got: <nil>: nil --- FAIL: TestOdhOperator/DSCInitialization_and_DataScienceCluster_validation_E2E_Tests (300.17s) === RUN TestOdhOperator/components/group_1/aigateway/Validate_component_enabled 2026-07-08T19:01:20Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.19.18"}} testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_1/aigateway/Validate_component_enabled (300.00s) === RUN TestOdhOperator/components/group_1/aigateway === PAUSE TestOdhOperator/components/group_1/aigateway === CONT TestOdhOperator/components/group_1/aigateway resource_applier_test.go:209: Timed out after 300.000s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:121 with: Expected resource 'default-dsc' of kind 'DataScienceCluster' to match condition {.spec.components.aigateway.managementState == "Removed" []}. Expected <string>: map[apiVersion:datasciencecluster.opendatahub.io/v2 kind:DataScienceCluster metadata:map[annotations:map[kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"datasciencecluster.opendatahub.io/v2","kind":"DataScienceCluster","metadata":{"annotations":{},"name":"default-dsc"},"spec":{"components":{"aipipelines":{"managementState":"Removed"},"codeflare":{"managementState":"Removed"},"dashboard":{"managementState":"Managed"},"kserve":{"defaultDeploymentMode":"RawDeployment","managementState":"Managed","serving":{"managementState":"Removed","name":"knative-serving"}},"kueue":{"managementState":"Removed"},"llamastackoperator":{"managementState":"Removed"},"mlflowoperator":{"managementState":"Removed"},"modelregistry":{"managementState":"Removed"},"ogx":{"managementState":"Removed"},"ray":{"managementState":"Removed"},"sparkoperator":{"managementState":"Removed"},"trainer":{"managementState":"Removed"},"trainingoperator":{"managementState":"Removed"},"trustyai":{"managementState":"Removed"},"workbenches":{"managementState":"Removed"}}}} ] creationTimestamp:2026-06-29T13:43:26Z generation:1954 managedFields:[map[apiVersion:datasciencecluster.opendatahub.io/v2 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:kubectl.kubernetes.io/last-applied-configuration:map[]]] f:status:map[f:components:map[f:aipipelines:map[f:managementState:map[] f:releases:map[k:{"name":"Kubeflow Pipelines"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:feastoperator:map[f:managementState:map[] f:releases:map[k:{"name":"Feast"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:kserve:map[f:managementState:map[] f:releases:map[k:{"name":"KServe"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-inference-scheduler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-workload-variant-autoscaler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"vLLM"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:kueue:map[f:managementState:map[]] f:llamastackoperator:map[] f:mlflowoperator:map[f:managementState:map[] f:releases:map[k:{"name":"MLflow"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:modelregistry:map[f:managementState:map[] f:registriesNamespace:map[] f:releases:map[k:{"name":"Kubeflow Model Registry"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"Open Data Hub Model Registry Operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:ogx:map[f:managementState:map[] f:releases:map[k:{"name":"OGX Operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"OGX"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:ray:map[f:managementState:map[] f:releases:map[k:{"name":"KubeRay"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:sparkoperator:map[f:managementState:map[] f:releases:map[k:{"name":"Spark Operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"Spark"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:trainer:map[f:managementState:map[] f:releases:map[k:{"name":"Kubeflow Trainer"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:trainingoperator:map[f:managementState:map[] f:releases:map[k:{"name":"Kubeflow Training Operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:trustyai:map[f:managementState:map[] f:releases:map[k:{"name":"EvalHub"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI Guardrails orchestrator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI LMEval driver"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI LMEval job"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI builtin detectors"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI service"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI sidecar gateway"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:workbenches:map[f:managementState:map[] f:releases:map[k:{"name":"Kubeflow Notebook Controller"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]] f:workbenchNamespace:map[]]] f:observedGeneration:map[] f:release:map[f:name:map[] f:version:map[]]]] manager:modules operation:Apply subresource:status time:2026-07-08T19:06:05Z] map[apiVersion:datasciencecluster.opendatahub.io/v2 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:kubectl.kubernetes.io/last-applied-configuration:map[]]] f:status:map[f:components:map[f:aipipelines:map[f:managementState:map[]] f:dashboard:map[f:managementState:map[]] f:feastoperator:map[f:managementState:map[]] f:kserve:map[f:managementState:map[] f:releases:map[k:{"name":"KServe"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-inference-scheduler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-workload-variant-autoscaler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"vLLM"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:kueue:map[f:managementState:map[]] f:llamastackoperator:map[] f:mlflowoperator:map[f:managementState:map[] f:releases:map[k:{"name":"MLflow"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:modelregistry:map[f:managementState:map[]] f:ogx:map[f:managementState:map[]] f:ray:map[f:managementState:map[]] f:sparkoperator:map[f:managementState:map[] f:releases:map[k:{"name":"Spark Operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"Spark"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:trainer:map[f:managementState:map[]] f:trainingoperator:map[f:managementState:map[]] f:trustyai:map[f:managementState:map[] f:releases:map[k:{"name":"EvalHub"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI Guardrails orchestrator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI LMEval driver"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI LMEval job"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI builtin detectors"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI service"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI sidecar gateway"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:workbenches:map[f:managementState:map[]]] f:conditions:map[] f:observedGeneration:map[] f:phase:map[] f:release:map[f:name:map[] f:version:map[]]]] manager:datasciencecluster operation:Apply subresource:status time:2026-07-08T19:06:11Z] map[apiVersion:datasciencecluster.opendatahub.io/v2 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[.:map[] f:kubectl.kubernetes.io/last-applied-configuration:map[]]] f:spec:map[.:map[] f:components:map[.:map[] f:aipipelines:map[] f:dashboard:map[] f:kserve:map[.:map[] f:rawDeploymentServiceConfig:map[]] f:kueue:map[.:map[] f:defaultClusterQueueName:map[] f:defaultLocalQueueName:map[]] f:llamastackoperator:map[] f:mlflowoperator:map[] f:modelregistry:map[.:map[] f:registriesNamespace:map[]] f:ogx:map[] f:ray:map[] f:sparkoperator:map[] f:trainer:map[] f:trainingoperator:map[] f:trustyai:map[.:map[] f:mcpGuardrailsMode:map[]] f:workbenches:map[.:map[] f:workbenchNamespace:map[]]]]] manager:kubectl-client-side-apply operation:Update time:2026-06-29T13:43:26Z] map[apiVersion:datasciencecluster.opendatahub.io/v2 fieldsType:FieldsV1 fieldsV1:map[f:spec:map[f:components:map[f:kserve:map[f:modelsAsService:map[.:map[] f:managementState:map[]]] f:llamastackoperator:map[f:managementState:map[]] f:mlflowoperator:map[f:managementState:map[]] f:trustyai:map[f:managementState:map[]]]]] manager:kubectl-patch operation:Update time:2026-07-08T18:13:22Z] map[apiVersion:datasciencecluster.opendatahub.io/v2 fieldsType:FieldsV1 fieldsV1:map[f:spec:map[f:components:map[f:aipipelines:map[f:managementState:map[]] f:dashboard:map[f:managementState:map[]] f:feastoperator:map[.:map[] f:managementState:map[]] f:kserve:map[f:managementState:map[] f:wva:map[.:map[] f:managementState:map[]]] f:kueue:map[f:managementState:map[]] f:modelregistry:map[f:managementState:map[]] f:ogx:map[f:managementState:map[]] f:ray:map[f:managementState:map[]] f:sparkoperator:map[f:managementState:map[]] f:trainer:map[f:managementState:map[]] f:trainingoperator:map[f:managementState:map[]] f:workbenches:map[f:managementState:map[]]]]] manager:e2e-tests operation:Update time:2026-07-08T19:06:05Z]] name:default-dsc resourceVersion:384612439 uid:191cdb15-01aa-4da4-9798-e6b508adbf9f] spec:map[components:map[aipipelines:map[managementState:Removed] dashboard:map[managementState:Removed] feastoperator:map[managementState:Removed] kserve:map[managementState:Managed modelsAsService:map[managementState:Managed] rawDeploymentServiceConfig:Headless wva:map[managementState:Removed]] kueue:map[defaultClusterQueueName:default defaultLocalQueueName:default managementState:Unmanaged] llamastackoperator:map[managementState:Removed] mlflowoperator:map[managementState:Managed] modelregistry:map[managementState:Removed registriesNamespace:rhoai-model-registries] ogx:map[managementState:Removed] ray:map[managementState:Removed] sparkoperator:map[managementState:Managed] trainer:map[managementState:Removed] trainingoperator:map[managementState:Removed] trustyai:map[managementState:Managed mcpGuardrailsMode:false] workbenches:map[managementState:Removed workbenchNamespace:rhods-notebooks]]] status:map[components:map[aipipelines:map[managementState:Removed releases:[map[name:Kubeflow Pipelines repoUrl:https://github.com/kubeflow/pipelines version:2.16.0]]] dashboard:map[managementState:Removed] feastoperator:map[managementState:Removed releases:[map[name:Feast repoUrl:https://github.com/feast-dev/feast version:0.63.0]]] kserve:map[managementState:Managed releases:[map[name:KServe repoUrl:https://github.com/kserve/kserve/ version:v0.17.0] map[name:vLLM repoUrl:https://github.com/vllm-project/vllm version:v0.18.0] map[name:llm-d-inference-scheduler repoUrl:https://github.com/llm-d/llm-d-inference-scheduler version:v0.7.1] map[name:llm-d-workload-variant-autoscaler repoUrl:https://github.com/llm-d/llm-d-workload-variant-autoscaler version:v0.6.0]]] kueue:map[managementState:Unmanaged] llamastackoperator:map[] mlflowoperator:map[managementState:Managed releases:[map[name:MLflow repoUrl:https://github.com/mlflow/mlflow version:v3.12.0]]] modelregistry:map[managementState:Removed registriesNamespace:rhoai-model-registries releases:[map[name:Kubeflow Model Registry repoUrl:https://github.com/kubeflow/model-registry version:latest] map[name:Open Data Hub Model Registry Operator repoUrl:https://github.com/opendatahub-io/model-registry-operator version:latest]]] ogx:map[managementState:Removed releases:[map[name:OGX repoUrl:https://github.com/opendatahub-io/ogx-distribution version:v1.0.2] map[name:OGX Operator repoUrl:https://github.com/opendatahub-io/ogx-k8s-operator version:v0.10.0]]] ray:map[managementState:Removed releases:[map[name:KubeRay repoUrl:https://github.com/ray-project/kuberay version:1.4.2]]] sparkoperator:map[managementState:Managed releases:[map[name:Spark Operator repoUrl:https://github.com/opendatahub-io/spark-operator version:v2.4.0] map[name:Spark repoUrl:https://github.com/apache/spark version:4.0.1]]] trainer:map[managementState:Removed releases:[map[name:Kubeflow Trainer repoUrl:https://github.com/kubeflow/trainer version:2.1.0]]] trainingoperator:map[managementState:Removed releases:[map[name:Kubeflow Training Operator repoUrl:https://github.com/kubeflow/trainer version:1.9.0]]] trustyai:map[managementState:Managed releases:[map[name:TrustyAI operator repoUrl:https://github.com/trustyai-explainability/trustyai-service-operator version:v1.37.0] map[name:TrustyAI service repoUrl:https://github.com/trustyai-explainability/trustyai-explainability version:v0.28.0] map[name:TrustyAI LMEval driver repoUrl:https://github.com/trustyai-explainability/trustyai-service-operator version:v1.37.0] map[name:TrustyAI LMEval job repoUrl:https://github.com/EleutherAI/lm-evaluation-harness version:v0.4.8] map[name:TrustyAI Guardrails orchestrator repoUrl:https://github.com/foundation-model-stack/fms-guardrails-orchestrator version:0.9.4] map[name:TrustyAI builtin detectors repoUrl:https://github.com/trustyai-explainability/guardrails-regex-detector version:v0.2.0] map[name:TrustyAI sidecar gateway repoUrl:https://github.com/trustyai-explainability/vllm-orchestrator-gateway version:v0.2.1] map[name:EvalHub repoUrl:https://github.com/eval-hub/eval-hub version:v0.3.0]]] workbenches:map[managementState:Removed releases:[map[name:Kubeflow Notebook Controller repoUrl:https://github.com/kubeflow/kubeflow version:1.10.0]] workbenchNamespace:rhods-notebooks]] conditions:[map[lastTransitionTime:2026-07-08T19:06:07Z status:True type:Ready] map[lastTransitionTime:2026-07-08T19:06:07Z observedGeneration:1954 status:True type:ProvisioningSucceeded] map[lastTransitionTime:2026-07-08T19:06:07Z status:True type:ComponentsReady] map[lastTransitionTime:2026-07-08T19:06:11Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:AIPipelinesReady] map[lastTransitionTime:2026-07-08T19:06:11Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:DashboardReady] map[lastTransitionTime:2026-07-08T19:06:11Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:FeastOperatorReady] map[lastTransitionTime:2026-07-08T19:02:05Z status:True type:KserveLLMInferenceServiceDependencies] map[lastTransitionTime:2026-07-08T19:02:05Z status:True type:KserveLLMInferenceServiceWideEPDependencies] map[lastTransitionTime:2026-07-08T19:06:11Z status:True type:KserveReady] map[lastTransitionTime:2026-07-08T19:06:11Z status:True type:KueueReady] map[lastTransitionTime:2026-07-08T19:06:11Z status:True type:MLflowOperatorReady] map[lastTransitionTime:2026-07-08T19:01:38Z message:MaaS gateway annotations and Authorino TLS are correctly configured reason:MaaSPrerequisitesMet status:True type:MaaSPrerequisitesAvailable] map[lastTransitionTime:2026-07-08T19:06:11Z status:True type:ModelControllerReady] map[lastTransitionTime:2026-07-08T19:06:11Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:ModelRegistryReady] map[lastTransitionTime:2026-07-08T19:06:11Z message:MaaS platform manifests applied and maas-api deployment is available reason:Reconciled status:True type:ModelsAsServiceReady] map[lastTransitionTime:2026-07-08T19:06:11Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:OGXReady] map[lastTransitionTime:2026-07-08T19:06:11Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:RayReady] map[lastTransitionTime:2026-07-08T19:06:11Z status:True type:SparkOperatorReady] map[lastTransitionTime:2026-07-08T19:06:11Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:TrainerReady] map[lastTransitionTime:2026-07-08T19:06:11Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:TrainingOperatorReady] map[lastTransitionTime:2026-07-08T19:06:11Z status:True type:TrustyAIReady] map[lastTransitionTime:2026-07-08T19:06:11Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:WorkbenchesReady]] observedGeneration:1954 phase:Ready release:map[name:OpenShift AI Self-Managed version:3.5.0-ea.2]]] to match expression <string>: .spec.components.aigateway.managementState == "Removed" --- FAIL: TestOdhOperator/components/group_1/aigateway (300.47s) === RUN TestOdhOperator/components/group_1/kserve/Validate_component_spec testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_1/kserve/Validate_component_spec (600.02s) === RUN TestOdhOperator/components/group_1/kserve === PAUSE TestOdhOperator/components/group_1/kserve === CONT TestOdhOperator/components/group_1/kserve test_context_test.go:223: Timed out after 600.001s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:121 with: Expected resource 'default-kserve' of kind 'Kserve' to match condition {.spec.nim.managementState == "" []}. Expected <string>: map[apiVersion:components.platform.opendatahub.io/v1alpha1 kind:Kserve metadata:map[annotations:map[component.opendatahub.io/management-state:Managed platform.opendatahub.io/instance.generation:1955 platform.opendatahub.io/instance.name:default-dsc platform.opendatahub.io/instance.uid:191cdb15-01aa-4da4-9798-e6b508adbf9f platform.opendatahub.io/type:OpenShift AI Self-Managed platform.opendatahub.io/version:3.5.0-ea.2] creationTimestamp:2026-07-08T19:01:40Z finalizers:[platform.opendatahub.io/finalizer] generation:1 labels:map[platform.opendatahub.io/part-of:datasciencecluster] managedFields:[map[apiVersion:components.platform.opendatahub.io/v1alpha1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:component.opendatahub.io/management-state:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:finalizers:map[v:"platform.opendatahub.io/finalizer":map[]] f:labels:map[f:platform.opendatahub.io/part-of:map[]] f:ownerReferences:map[k:{"uid":"191cdb15-01aa-4da4-9798-e6b508adbf9f"}:map[]]] f:status:map[f:conditions:map[] f:observedGeneration:map[] f:phase:map[] f:releases:map[k:{"name":"KServe"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-inference-scheduler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-workload-variant-autoscaler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"vLLM"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]]] manager:kserve operation:Apply subresource:status time:2026-07-08T19:04:40Z] map[apiVersion:components.platform.opendatahub.io/v1alpha1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:component.opendatahub.io/management-state:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:labels:map[f:platform.opendatahub.io/part-of:map[]] f:ownerReferences:map[k:{"uid":"191cdb15-01aa-4da4-9798-e6b508adbf9f"}:map[]]] f:spec:map[f:modelsAsService:map[f:managementState:map[]] f:nim:map[] f:rawDeploymentServiceConfig:map[] f:wva:map[f:managementState:map[]]]] manager:datasciencecluster operation:Apply time:2026-07-08T19:06:50Z] map[apiVersion:components.platform.opendatahub.io/v1alpha1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:finalizers:map[.:map[] v:"platform.opendatahub.io/finalizer":map[]]]] manager:manager operation:Update time:2026-07-08T19:01:40Z]] name:default-kserve ownerReferences:[map[apiVersion:datasciencecluster.opendatahub.io/v2 blockOwnerDeletion:true controller:true kind:DataScienceCluster name:default-dsc uid:191cdb15-01aa-4da4-9798-e6b508adbf9f]] resourceVersion:384613080 uid:5b940f50-d2b1-47dd-bc84-46d30a6fd82b] spec:map[modelsAsService:map[managementState:Managed] nim:map[airGapped:false managementState:Managed] rawDeploymentServiceConfig:Headless wva:map[managementState:Removed]] status:map[conditions:[map[lastTransitionTime:2026-07-08T19:02:38Z status:True type:Ready] map[lastTransitionTime:2026-07-08T19:02:38Z observedGeneration:1 status:True type:ProvisioningSucceeded] map[lastTransitionTime:2026-07-08T19:02:40Z observedGeneration:1 status:True type:DeploymentsAvailable] map[lastTransitionTime:2026-07-08T19:02:04Z observedGeneration:1 status:True type:DependenciesAvailable] map[lastTransitionTime:2026-07-08T19:02:04Z observedGeneration:1 status:True type:KserveLLMInferenceServiceDependencies] map[lastTransitionTime:2026-07-08T19:02:04Z observedGeneration:1 status:True type:KserveLLMInferenceServiceWideEPDependencies]] observedGeneration:1 phase:Ready releases:[map[name:KServe repoUrl:https://github.com/kserve/kserve/ version:v0.17.0] map[name:vLLM repoUrl:https://github.com/vllm-project/vllm version:v0.18.0] map[name:llm-d-inference-scheduler repoUrl:https://github.com/llm-d/llm-d-inference-scheduler version:v0.7.1] map[name:llm-d-workload-variant-autoscaler repoUrl:https://github.com/llm-d/llm-d-workload-variant-autoscaler version:v0.6.0]]]] to match expression <string>: .spec.nim.managementState == "" --- FAIL: TestOdhOperator/components/group_1/kserve (682.13s) === RUN TestOdhOperator/components/group_1 --- FAIL: TestOdhOperator/components/group_1 (0.00s) === RUN TestOdhOperator/components/group_2/kueue/Validate_autoCreateQueues_disabled_skips_queue_creation helper_test.go:156: Setting Kueue component (default-kueue) to Removed mode to start with a clean state. helper_test.go:156: Cleaning up any existing Kueue test resources. helper_test.go:156: Cleaning up Kueue cluster-scoped resources helper_test.go:156: Will attempt to delete 3 Kueue resources helper_test.go:156: Attempting to delete ClusterQueue /default helper_test.go:156: Successfully processed deletion of ClusterQueue /default helper_test.go:156: Attempting to delete ResourceFlavor /default-flavor helper_test.go:156: Successfully processed deletion of ResourceFlavor /default-flavor helper_test.go:156: Attempting to delete Kueue /cluster helper_test.go:156: Successfully processed deletion of Kueue /cluster helper_test.go:156: Ensuring LWS Operator is installed (namespace=openshift-lws-operator, name=leader-worker-set). helper_test.go:156: Ensuring LeaderWorkerSetOperator CR exists (namespace=openshift-lws-operator, name=cluster). helper_test.go:156: Ensuring OCP Kueue Operator is installed (namespace=openshift-kueue-operator, name=kueue-operator). helper_test.go:156: Creating test namespace (test-kueue-no-queues-d97a2vfmmugs7b0pljl0) with Kueue management annotation. helper_test.go:156: Creating Kueue ConfigMap (kueue-manager-config) to verify default Kueue resource is created correctly. helper_test.go:156: Creating Kueue ConfigMap (kueue-manager-config) in namespace (redhat-ods-applications). helper_test.go:156: Updating DSC to set Kueue to Unmanaged with autoCreateQueues=false (default). 2026-07-08T19:14:15Z INFO unknown field "spec.components.kueue.autoCreateQueues" 2026-07-08T19:14:25Z INFO unknown field "spec.components.kueue.autoCreateQueues" helper_test.go:156: Verifying Kueue configuration (cluster) is still created. helper_test.go:156: Verifying no ClusterQueue, ResourceFlavor or LocalQueue is created (autoCreateQueues=false). testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_2/kueue/Validate_autoCreateQueues_disabled_skips_queue_creation (320.52s) === RUN TestOdhOperator/components/group_2/kueue === PAUSE TestOdhOperator/components/group_2/kueue === CONT TestOdhOperator/components/group_2/kueue test_context_test.go:1106: cleanup: best-effort delete failed for /, Kind= redhat-ods-applications/kueue-manager-config: Object 'Kind' is missing in 'unstructured object has no kind' test_context_test.go:615: Timed out after 300.008s. The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1824 with: Expected resource list 'default' of kind 'ClusterQueue' to be empty, but it contains resources. Expected <[]unstructured.Unstructured | len:1, cap:1>: [ { Object: { "apiVersion": <string>"kueue.x-k8s.io/v1beta2", "kind": <string>"ClusterQueue", "metadata": <map[string]interface {} | len:8>{ "managedFields": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:6>{ "apiVersion": <string>"kueue.x-k8s.io/v1beta2", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:2>{ "f:spec": <map[string]interface {} | len:7>{ "f:queueingStrategy": <map[string]interface {} | len:0>{}, "f:resourceGroups": <map[string]interface {} | len:0>{}, "f:stopPolicy": <map[string]interface {} | len:0>{}, ".": <map[string]interface {} | len:0>{}, "f:flavorFungibility": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:whenCanBorrow": <map[string]interface {} | len:0>{}, "f:whenCanPreempt": <map[string]interface {} | len:0>{}, }, "f:namespaceSelector": <map[string]interface {} | len:0>{}, "f:preemption": <map[string]interface {} | len:4>{ "f:borrowWithinCohort": <map[string]interface {} | len:2>{...: ..., ...: ...}, "f:reclaimWithinCohort": <map[string]interface {} | len:0>{}, "f:withinClusterQueue": <map[string]interface {} | len:0>{}, ".": <map[string]interface {} | len:0>{}, }, }, "f:metadata": <map[string]interface {} | len:2>{ "f:labels": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/part-of": <map[string]interface {} | len:0>{}, }, "f:annotations": <map[string]interface {} | len:6>{ ".": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/instance.generation": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/instance.name": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/instance.uid": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/type": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/version": <map[string]interface {} | len:0>{}, }, }, }, "manager": <string>"manager", "operation": <string>"Update", "time": <string>"2026-07-08T19:14:17Z", }, ], "name": <string>"default", "resourceVersion": <string>"384620115", "uid": <string>"2c16a0ec-715a-47b3-9750-7019091c4d66", "annotations": <map[string]interface {} | len:5>{ "platform.opendatahub.io/instance.generation": <string>"1", "platform.opendatahub.io/instance.name": <string>"default-kueue", "platform.opendatahub.io/instance.uid": <string>"3386bcd2-883d-4913-b7ec-7cfc7af669f6", "platform.opendatahub.io/type": <string>"OpenShift AI Self-Managed", "platform.opendatahub.io/version": <string>"3.5.0-ea.2", }, "creationTimestamp": <string>"2026-07-08T19:14:17Z", "generation": <int64>1, "labels": <map[string]interface {} | len:1>{ "platform.opendatahub.io/part-of": <string>"kueue", }, }, "spec": <map[string]interface {} | len:6>{ "resourceGroups": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:2>{ "flavors": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:2>{ "resources": <[]interface {} | len:2, cap:2>[ <map[string]interface {} | len:2>{...: ..., ...: ...}, <map[string]interface {} | len:2>{...: ..., ...: ...}, ], "name": <string>"default-flavor", }, ], "coveredResources": <[]interface {} | len:2, cap:2>[<string>"cpu", <string>"memory"], }, ], "stopPolicy": <string>"None", "flavorFungibility": <map[string]interface {} | len:2>{ "whenCanBorrow": <string>"MayStopSearch", "whenCanPreempt": <string>"TryNextFlavor", }, "namespaceSelector": <map[string]interface {} | len:1>{ "matchLabels": <map[string]interface {} | len:1>{ "kueue.openshift.io/managed": <string>"true", }, }, "preemption": <map[string]interface {} | len:3>{ "borrowWithinCohort": <map[string]interface {} | len:1>{ "policy": <string>"Never", }, "reclaimWithinCohort": <string>"Never", "withinClusterQueue": <string>"Never", }, "queueingStrategy": <string>"BestEffortFIFO", }, }, }, ] to be empty test_context_test.go:1106: cleanup: best-effort delete failed for /, Kind= redhat-ods-applications/kueue-manager-config: Object 'Kind' is missing in 'unstructured object has no kind' --- FAIL: TestOdhOperator/components/group_2/kueue (402.65s) === RUN TestOdhOperator/components/group_2 --- FAIL: TestOdhOperator/components/group_2 (0.00s) === RUN TestOdhOperator/components 2026/07/08 19:12:42 TEST FAILURE DETECTED: group 1 2026/07/08 19:12:42 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 1) === 2026/07/08 19:12:43 === CLUSTER STATE === 2026/07/08 19:12:43 Node: ods-qe-psi-07-b4mqp-master-0 2026/07/08 19:12:43 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:12:43 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:12:43 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:12:43 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:12:43 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/08 19:12:43 Node: ods-qe-psi-07-b4mqp-master-1 2026/07/08 19:12:43 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:12:43 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:12:43 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:12:43 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:12:43 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/08 19:12:43 Node: ods-qe-psi-07-b4mqp-master-2 2026/07/08 19:12:43 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:12:43 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:12:43 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:12:43 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:12:43 Resources: 11500m CPU, 23450732Ki memory allocatable (of 12000m CPU, 24601708Ki memory capacity) 2026/07/08 19:12:43 Node: ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/08 19:12:43 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:12:43 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:12:43 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:12:43 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:12:43 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/08 19:12:43 Node: ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 19:12:43 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:12:43 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:12:43 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:12:43 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:12:43 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/08 19:12:43 Node: ods-qe-psi-07-b4mqp-worker-0-jd4pm 2026/07/08 19:12:43 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:12:43 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:12:43 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:12:43 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:12:43 Resources: 11500m CPU, 23452736Ki memory allocatable (of 12000m CPU, 24603712Ki memory capacity) 2026/07/08 19:12:43 Node: ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/08 19:12:43 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:12:43 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:12:43 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:12:43 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:12:43 Resources: 11500m CPU, 23452744Ki memory allocatable (of 12000m CPU, 24603720Ki memory capacity) 2026/07/08 19:12:43 Node: ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/08 19:12:43 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:12:43 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:12:43 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:12:43 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:12:43 Resources: 11500m CPU, 23450720Ki memory allocatable (of 12000m CPU, 24601696Ki memory capacity) 2026/07/08 19:12:43 === DEPLOYMENTS === 2026/07/08 19:12:43 Namespace: kube-system 2026/07/08 19:12:43 Namespace: redhat-ods-applications 2026/07/08 19:12:43 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/08 19:12:43 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/08 19:12:43 Deployment maas-api: 1/1 (READY) 2026/07/08 19:12:43 Deployment maas-controller: 1/1 (READY) 2026/07/08 19:12:43 Deployment maas-postgres: 1/1 (READY) 2026/07/08 19:12:43 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/08 19:12:43 Deployment model-serving-api: 1/1 (READY) 2026/07/08 19:12:43 Deployment odh-model-controller: 1/1 (READY) 2026/07/08 19:12:43 Deployment postgres: 1/1 (READY) 2026/07/08 19:12:43 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/08 19:12:43 === PODS === 2026/07/08 19:12:43 Failed to collect pod data: redhat-ods-applications/trustyai-service-operator-controller-manager-6f7f88c874-sgx9v: container manager restarts=2 2026/07/08 19:12:43 === OPERATOR STATUS === 2026/07/08 19:12:43 Operator deployment: 3/3 ready 2026/07/08 19:12:43 === DSCI STATUS === 2026/07/08 19:12:43 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/08 19:12:43 === DSC STATUS === 2026/07/08 19:12:43 DSC default-dsc: 2026/07/08 19:12:43 Ready: True - 2026/07/08 19:12:43 ProvisioningSucceeded: True - 2026/07/08 19:12:43 ComponentsReady: True - 2026/07/08 19:12:43 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/08 19:12:43 DashboardReady: False - Component ManagementState is set to Removed 2026/07/08 19:12:43 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/08 19:12:43 KserveLLMInferenceServiceDependencies: True - 2026/07/08 19:12:43 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/08 19:12:43 KserveReady: True - 2026/07/08 19:12:43 KueueReady: True - 2026/07/08 19:12:43 MLflowOperatorReady: True - 2026/07/08 19:12:43 MaaSPrerequisitesAvailable: True - MaaS gateway annotations and Authorino TLS are correctly configured 2026/07/08 19:12:43 ModelControllerReady: True - 2026/07/08 19:12:43 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/08 19:12:43 ModelsAsServiceReady: True - MaaS platform manifests applied and maas-api deployment is available 2026/07/08 19:12:43 OGXReady: False - Component ManagementState is set to Removed 2026/07/08 19:12:43 RayReady: False - Component ManagementState is set to Removed 2026/07/08 19:12:43 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/08 19:12:43 TrainerReady: False - Component ManagementState is set to Removed 2026/07/08 19:12:43 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/08 19:12:43 TrustyAIReady: True - 2026/07/08 19:12:43 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/08 19:12:43 === RECENT EVENTS === 2026/07/08 19:12:43 19:09:31 WARN Pod/maas-controller-656b6867cb-slxjq: Unhealthy - Readiness probe failed: Get "http://10.130.2.235:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026/07/08 19:12:43 === RESOURCE QUOTAS === 2026/07/08 19:12:43 No resource quotas found in monitored namespaces 2026/07/08 19:12:43 Classification for group 1: infrastructure/probe-failure (code=1014, confidence=medium) 2026/07/08 19:12:43 Evidence: event Pod/maas-controller-656b6867cb-slxjq: Unhealthy - Readiness probe failed: Get "http://10.130.2.235:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"probe-failure","error_code":1014,"evidence":["event Pod/maas-controller-656b6867cb-slxjq: Unhealthy - Readiness probe failed: Get \"http://10.130.2.235:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"],"confidence":"medium"} 2026/07/08 19:12:43 Diagnostics complete for failed test: group 1 controller_test.go:763: Stopping: group 1 test failed. 2026/07/08 19:12:43 Circuit breaker: failure classified as infrastructure/probe-failure (code=1014) 2026/07/08 19:12:43 Circuit breaker: infrastructure/unknown failure recorded (2/3) 2026/07/08 19:19:25 TEST FAILURE DETECTED: group 2 2026/07/08 19:19:25 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 2) === 2026/07/08 19:19:26 === CLUSTER STATE === 2026/07/08 19:19:26 Node: ods-qe-psi-07-b4mqp-master-0 2026/07/08 19:19:26 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:19:26 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:19:26 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:19:26 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:19:26 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/08 19:19:26 Node: ods-qe-psi-07-b4mqp-master-1 2026/07/08 19:19:26 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:19:26 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:19:26 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:19:26 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:19:26 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/08 19:19:26 Node: ods-qe-psi-07-b4mqp-master-2 2026/07/08 19:19:26 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:19:26 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:19:26 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:19:26 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:19:26 Resources: 11500m CPU, 23450732Ki memory allocatable (of 12000m CPU, 24601708Ki memory capacity) 2026/07/08 19:19:26 Node: ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/08 19:19:26 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:19:26 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:19:26 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:19:26 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:19:26 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/08 19:19:26 Node: ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 19:19:26 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:19:26 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:19:26 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:19:26 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:19:26 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/08 19:19:26 Node: ods-qe-psi-07-b4mqp-worker-0-jd4pm 2026/07/08 19:19:26 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:19:26 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:19:26 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:19:26 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:19:26 Resources: 11500m CPU, 23452736Ki memory allocatable (of 12000m CPU, 24603712Ki memory capacity) 2026/07/08 19:19:26 Node: ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/08 19:19:26 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:19:26 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:19:26 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:19:26 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:19:26 Resources: 11500m CPU, 23452744Ki memory allocatable (of 12000m CPU, 24603720Ki memory capacity) 2026/07/08 19:19:26 Node: ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/08 19:19:26 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:19:26 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:19:26 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:19:26 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:19:26 Resources: 11500m CPU, 23450720Ki memory allocatable (of 12000m CPU, 24601696Ki memory capacity) 2026/07/08 19:19:26 === DEPLOYMENTS === 2026/07/08 19:19:26 Namespace: redhat-ods-applications 2026/07/08 19:19:26 Deployment maas-postgres: 1/1 (READY) 2026/07/08 19:19:26 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/08 19:19:26 Deployment postgres: 1/1 (READY) 2026/07/08 19:19:26 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/08 19:19:26 Namespace: kube-system 2026/07/08 19:19:26 === PODS === 2026/07/08 19:19:26 Failed to collect pod data: redhat-ods-applications/trustyai-service-operator-controller-manager-6f7f88c874-sgx9v: container manager restarts=2 2026/07/08 19:19:26 === OPERATOR STATUS === 2026/07/08 19:19:26 Operator deployment: 3/3 ready 2026/07/08 19:19:26 === DSCI STATUS === 2026/07/08 19:19:26 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/08 19:19:26 === DSC STATUS === 2026/07/08 19:19:26 Failed to collect DSC data: default-dsc unhealthy conditions: ModelsAsServiceReady=False 2026/07/08 19:19:26 === RECENT EVENTS === 2026/07/08 19:19:26 19:15:57 INFO Pod/maas-api-84c4b8dcb6-kbczb: Killing - Stopping container maas-api 2026/07/08 19:19:26 19:15:57 INFO Pod/model-registry-operator-controller-manager-5f68f4fd76-nqsnq: Killing - Stopping container manager 2026/07/08 19:19:26 19:15:57 INFO Pod/maas-controller-656b6867cb-slxjq: Killing - Stopping container manager 2026/07/08 19:19:26 19:15:57 INFO Pod/odh-model-controller-6b647f87c-zw8fd: Killing - Stopping container manager 2026/07/08 19:19:26 19:15:56 INFO Pod/model-serving-api-78bbbb9bf8-62pmv: Killing - Stopping container server 2026/07/08 19:19:26 19:15:56 INFO Pod/workload-variant-autoscaler-controller-manager-56865c9848-ktqss: Killing - Stopping container manager 2026/07/08 19:19:26 19:15:55 INFO Pod/llmisvc-controller-manager-586c85c465-wj6m7: Killing - Stopping container manager 2026/07/08 19:19:26 19:15:55 INFO Pod/kserve-controller-manager-6c8cb86d7f-knwkb: Killing - Stopping container manager 2026/07/08 19:19:26 19:15:47 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - ServiceMonitor controller-manager-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/08 19:19:26 19:15:42 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6b647f87c-zw8fd_9feba9fb-44bb-4ce7-9796-3319212316e3 became leader 2026/07/08 19:19:26 19:15:41 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-56865c9848-ktqss_e0977608-0bfe-4b73-9834-c8473d110cf6 became leader 2026/07/08 19:19:26 19:15:40 INFO Pod/workload-variant-autoscaler-controller-manager-56865c9848-ktqss: Started - Started container manager 2026/07/08 19:19:26 19:15:40 INFO Pod/workload-variant-autoscaler-controller-manager-56865c9848-ktqss: Created - Created container: manager 2026/07/08 19:19:26 19:15:40 INFO Pod/workload-variant-autoscaler-controller-manager-56865c9848-ktqss: Pulled - Container image "registry.redhat.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:3d219851cf91eb92db3df70276a7e0615e56f41deeff9a7b83b5e93b686bb789" already present on machine 2026/07/08 19:19:26 19:15:40 INFO Pod/workload-variant-autoscaler-controller-manager-56865c9848-ktqss: AddedInterface - Add eth0 [10.131.0.20/23] from ovn-kubernetes 2026/07/08 19:19:26 19:15:39 INFO Pod/workload-variant-autoscaler-controller-manager-56865c9848-ktqss: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-56865c9848-ktqss to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/08 19:19:26 19:15:39 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-56865c9848: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-56865c9848-ktqss 2026/07/08 19:19:26 19:15:39 INFO Pod/workload-variant-autoscaler-controller-manager-56865c9848-z86bw: Killing - Stopping container manager 2026/07/08 19:19:26 19:15:39 WARN Pod/workload-variant-autoscaler-controller-manager-56865c9848-z86bw: Unhealthy - Readiness probe failed: Get "http://10.129.3.16:8081/readyz": dial tcp 10.129.3.16:8081: connect: connection refused 2026/07/08 19:19:26 19:15:39 INFO Pod/workload-variant-autoscaler-controller-manager-56865c9848-dswg5: Killing - Stopping container manager 2026/07/08 19:19:26 19:15:39 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-56865c9848 from 0 to 1 2026/07/08 19:19:26 19:15:26 INFO Pod/odh-model-controller-6b647f87c-zw8fd: Started - Started container manager 2026/07/08 19:19:26 19:15:26 INFO Pod/odh-model-controller-6b647f87c-zw8fd: Created - Created container: manager 2026/07/08 19:19:26 19:15:26 INFO Pod/odh-model-controller-6b647f87c-zw8fd: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b423702816000ce0f075548e3a67de006f40ccf05c575a866cb62e2ac9711191" in 857ms (857ms including waiting). Image size: 238719956 bytes. 2026/07/08 19:19:26 19:15:25 INFO Pod/odh-model-controller-6b647f87c-zw8fd: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b423702816000ce0f075548e3a67de006f40ccf05c575a866cb62e2ac9711191" 2026/07/08 19:19:26 19:15:25 INFO Pod/odh-model-controller-6b647f87c-zw8fd: AddedInterface - Add eth0 [10.129.3.22/23] from ovn-kubernetes 2026/07/08 19:19:26 19:15:24 INFO Pod/odh-model-controller-6b647f87c-zw8fd: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6b647f87c-zw8fd to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 19:19:26 19:15:24 INFO Pod/odh-model-controller-6b647f87c-w8mrt: Killing - Stopping container manager 2026/07/08 19:19:26 19:15:24 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6b647f87c from 0 to 1 2026/07/08 19:19:26 19:15:24 INFO Pod/odh-model-controller-6b647f87c-bg5ln: Killing - Stopping container manager 2026/07/08 19:19:26 19:15:24 INFO ReplicaSet/odh-model-controller-6b647f87c: SuccessfulCreate - Created pod: odh-model-controller-6b647f87c-zw8fd 2026/07/08 19:19:26 19:15:10 INFO Pod/model-serving-api-78bbbb9bf8-62pmv: Started - Started container server 2026/07/08 19:19:26 19:15:10 INFO Pod/model-serving-api-78bbbb9bf8-62pmv: Created - Created container: server 2026/07/08 19:19:26 19:15:10 INFO Pod/model-serving-api-78bbbb9bf8-62pmv: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:6dd699e678c3f4d6e51138eea62f096f26472b5f9c1ce2757cf9e9e2f80371a4" already present on machine 2026/07/08 19:19:26 19:15:10 INFO Pod/model-serving-api-78bbbb9bf8-62pmv: AddedInterface - Add eth0 [10.130.3.5/23] from ovn-kubernetes 2026/07/08 19:19:26 19:15:09 INFO Pod/model-serving-api-78bbbb9bf8-62pmv: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-78bbbb9bf8-62pmv to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/08 19:19:26 19:15:09 INFO ReplicaSet/model-serving-api-78bbbb9bf8: SuccessfulCreate - Created pod: model-serving-api-78bbbb9bf8-62pmv 2026/07/08 19:19:26 19:15:09 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-78bbbb9bf8 from 0 to 1 2026/07/08 19:19:26 19:15:09 INFO Pod/model-serving-api-78bbbb9bf8-nlxnv: Killing - Stopping container server 2026/07/08 19:19:26 19:15:09 INFO Pod/model-serving-api-78bbbb9bf8-mhs8z: Killing - Stopping container server 2026/07/08 19:19:26 19:15:04 INFO CronJob/maas-api-key-cleanup: SawCompletedJob - Saw completed job: maas-api-key-cleanup-29725635, condition: Complete 2026/07/08 19:19:26 19:15:04 INFO Job/maas-api-key-cleanup-29725635: Completed - Job completed 2026/07/08 19:19:26 19:15:01 INFO Pod/maas-api-key-cleanup-29725635-mkmkq: Started - Started container cleanup 2026/07/08 19:19:26 19:15:01 INFO Pod/maas-api-key-cleanup-29725635-mkmkq: Created - Created container: cleanup 2026/07/08 19:19:26 19:15:00 INFO Pod/maas-api-key-cleanup-29725635-mkmkq: Scheduled - Successfully assigned redhat-ods-applications/maas-api-key-cleanup-29725635-mkmkq to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 19:19:26 19:15:00 INFO CronJob/maas-api-key-cleanup: SuccessfulCreate - Created job maas-api-key-cleanup-29725635 2026/07/08 19:19:26 19:15:00 INFO Job/maas-api-key-cleanup-29725635: SuccessfulCreate - Created pod: maas-api-key-cleanup-29725635-mkmkq 2026/07/08 19:19:26 19:15:00 INFO Pod/maas-api-key-cleanup-29725635-mkmkq: Pulled - Container image "registry.redhat.io/ubi9/ubi-minimal@sha256:907b68736aa798b2d38255b7aa070b2a70acb90803864a40f05d0ec47556ddd0" already present on machine 2026/07/08 19:19:26 19:15:00 INFO Pod/maas-api-key-cleanup-29725635-mkmkq: AddedInterface - Add eth0 [10.129.3.19/23] from ovn-kubernetes 2026/07/08 19:19:26 === RESOURCE QUOTAS === 2026/07/08 19:19:26 No resource quotas found in monitored namespaces 2026/07/08 19:19:26 Classification for group 2: infrastructure/probe-failure (code=1014, confidence=medium) 2026/07/08 19:19:26 Evidence: event Pod/workload-variant-autoscaler-controller-manager-56865c9848-z86bw: Unhealthy - Readiness probe failed: Get "http://10.129.3.16:8081/readyz": dial tcp 10.129.3.16:8081: connect: connection refused FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"probe-failure","error_code":1014,"evidence":["event Pod/workload-variant-autoscaler-controller-manager-56865c9848-z86bw: Unhealthy - Readiness probe failed: Get \"http://10.129.3.16:8081/readyz\": dial tcp 10.129.3.16:8081: connect: connection refused"],"confidence":"medium"} 2026/07/08 19:19:26 Diagnostics complete for failed test: group 2 controller_test.go:763: Stopping: group 2 test failed. 2026/07/08 19:19:26 Circuit breaker: failure classified as infrastructure/probe-failure (code=1014) 2026/07/08 19:19:26 Circuit breaker: infrastructure/unknown failure recorded (3/3) 2026/07/08 19:19:26 Circuit breaker: threshold reached (3 failures), running confirming health check... 2026/07/08 19:19:26 Circuit breaker: confirming health check passed — cluster appears healthy, resetting counter --- FAIL: TestOdhOperator/components (1538.11s) === RUN TestOdhOperator 2026/07/08 18:56:08 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/08 19:01:19 TEST FAILURE DETECTED: DSCInitialization and DataScienceCluster validation E2E Tests 2026/07/08 19:01:19 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:DSCInitialization and DataScienceCluster validation E2E Tests) === 2026/07/08 19:01:20 === CLUSTER STATE === 2026/07/08 19:01:20 Node: ods-qe-psi-07-b4mqp-master-0 2026/07/08 19:01:20 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:01:20 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:01:20 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:01:20 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:01:20 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/08 19:01:20 Node: ods-qe-psi-07-b4mqp-master-1 2026/07/08 19:01:20 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:01:20 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:01:20 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:01:20 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:01:20 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/08 19:01:20 Node: ods-qe-psi-07-b4mqp-master-2 2026/07/08 19:01:20 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:01:20 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:01:20 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:01:20 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:01:20 Resources: 11500m CPU, 23450732Ki memory allocatable (of 12000m CPU, 24601708Ki memory capacity) 2026/07/08 19:01:20 Node: ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/08 19:01:20 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:01:20 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:01:20 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:01:20 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:01:20 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/08 19:01:20 Node: ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 19:01:20 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:01:20 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:01:20 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:01:20 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:01:20 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/08 19:01:20 Node: ods-qe-psi-07-b4mqp-worker-0-jd4pm 2026/07/08 19:01:20 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:01:20 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:01:20 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:01:20 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:01:20 Resources: 11500m CPU, 23452736Ki memory allocatable (of 12000m CPU, 24603712Ki memory capacity) 2026/07/08 19:01:20 Node: ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/08 19:01:20 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:01:20 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:01:20 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:01:20 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:01:20 Resources: 11500m CPU, 23452744Ki memory allocatable (of 12000m CPU, 24603720Ki memory capacity) 2026/07/08 19:01:20 Node: ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/08 19:01:20 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:01:20 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:01:20 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:01:20 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:01:20 Resources: 11500m CPU, 23450720Ki memory allocatable (of 12000m CPU, 24601696Ki memory capacity) 2026/07/08 19:01:20 === DEPLOYMENTS === 2026/07/08 19:01:20 Namespace: redhat-ods-applications 2026/07/08 19:01:20 Deployment dashboard-redirect: 2/2 (READY) 2026/07/08 19:01:20 Deployment data-science-pipelines-operator-controller-manager: 1/1 (READY) 2026/07/08 19:01:20 Deployment feast-operator-controller-manager: 1/1 (READY) 2026/07/08 19:01:20 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/08 19:01:20 Deployment kubeflow-trainer-controller-manager: 1/1 (READY) 2026/07/08 19:01:20 Deployment kubeflow-training-operator: 1/1 (READY) 2026/07/08 19:01:20 Deployment kuberay-operator: 1/1 (READY) 2026/07/08 19:01:20 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/08 19:01:20 Deployment maas-api: 1/1 (READY) 2026/07/08 19:01:20 Deployment maas-controller: 1/1 (READY) 2026/07/08 19:01:20 Deployment maas-postgres: 1/1 (READY) 2026/07/08 19:01:20 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/08 19:01:20 Deployment model-registry-operator-controller-manager: 1/1 (READY) 2026/07/08 19:01:20 Deployment model-serving-api: 1/1 (READY) 2026/07/08 19:01:20 Deployment notebook-controller-deployment: 1/1 (READY) 2026/07/08 19:01:20 Deployment odh-model-controller: 1/1 (READY) 2026/07/08 19:01:20 Deployment odh-notebook-controller-manager: 1/1 (READY) 2026/07/08 19:01:20 Deployment ogx-k8s-operator-controller-manager: 1/1 (READY) 2026/07/08 19:01:20 Deployment postgres: 1/1 (READY) 2026/07/08 19:01:20 Deployment rhods-dashboard: 2/2 (READY) 2026/07/08 19:01:20 Deployment spark-operator-controller: 1/1 (READY) 2026/07/08 19:01:20 Deployment spark-operator-webhook: 1/1 (READY) 2026/07/08 19:01:20 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/08 19:01:20 Namespace: kube-system 2026/07/08 19:01:20 === PODS === 2026/07/08 19:01:20 Failed to collect pod data: redhat-ods-applications/maas-controller-656b6867cb-x6k8q: container manager restarts=14; redhat-ods-applications/trustyai-service-operator-controller-manager-6f7f88c874-sgx9v: container manager restarts=2 2026/07/08 19:01:20 === OPERATOR STATUS === 2026/07/08 19:01:20 Operator deployment: 3/3 ready 2026/07/08 19:01:20 === DSCI STATUS === 2026/07/08 19:01:20 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/08 19:01:20 === DSC STATUS === 2026/07/08 19:01:20 DSC default-dsc: 2026/07/08 19:01:20 Ready: True - 2026/07/08 19:01:20 ProvisioningSucceeded: True - 2026/07/08 19:01:20 ModulesReady: True - 2026/07/08 19:01:20 AIPipelinesReady: True - 2026/07/08 19:01:20 ComponentsReady: True - 2026/07/08 19:01:20 DashboardReady: True - 2026/07/08 19:01:20 FeastOperatorReady: True - 2026/07/08 19:01:20 KserveLLMInferenceServiceDependencies: True - 2026/07/08 19:01:20 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/08 19:01:20 KserveReady: True - 2026/07/08 19:01:20 KueueReady: True - 2026/07/08 19:01:20 MLflowOperatorReady: True - 2026/07/08 19:01:20 MaaSPrerequisitesAvailable: True - MaaS gateway annotations and Authorino TLS are correctly configured 2026/07/08 19:01:20 ModelControllerReady: True - 2026/07/08 19:01:20 ModelRegistryReady: True - 2026/07/08 19:01:20 ModelsAsServiceReady: True - MaaS platform manifests applied and maas-api deployment is available 2026/07/08 19:01:20 OGXReady: True - 2026/07/08 19:01:20 RayReady: True - 2026/07/08 19:01:20 SparkOperatorReady: True - 2026/07/08 19:01:20 TrainerReady: True - 2026/07/08 19:01:20 TrainingOperatorReady: True - 2026/07/08 19:01:20 TrustyAIReady: True - 2026/07/08 19:01:20 WorkbenchesReady: True - 2026/07/08 19:01:20 === RECENT EVENTS === 2026/07/08 19:01:20 19:00:05 INFO Job/maas-api-key-cleanup-29725620: Completed - Job completed 2026/07/08 19:01:20 19:00:05 INFO CronJob/maas-api-key-cleanup: SuccessfulDelete - Deleted job maas-api-key-cleanup-29725575 2026/07/08 19:01:20 19:00:05 INFO CronJob/maas-api-key-cleanup: SawCompletedJob - Saw completed job: maas-api-key-cleanup-29725620, condition: Complete 2026/07/08 19:01:20 19:00:01 INFO Pod/maas-api-key-cleanup-29725620-spctg: Created - Created container: cleanup 2026/07/08 19:01:20 19:00:01 INFO Pod/maas-api-key-cleanup-29725620-spctg: Started - Started container cleanup 2026/07/08 19:01:20 19:00:00 INFO Pod/maas-api-key-cleanup-29725620-spctg: Scheduled - Successfully assigned redhat-ods-applications/maas-api-key-cleanup-29725620-spctg to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/08 19:01:20 19:00:00 INFO Pod/maas-api-key-cleanup-29725620-spctg: AddedInterface - Add eth0 [10.130.2.230/23] from ovn-kubernetes 2026/07/08 19:01:20 19:00:00 INFO Pod/maas-api-key-cleanup-29725620-spctg: Pulled - Container image "registry.redhat.io/ubi9/ubi-minimal@sha256:907b68736aa798b2d38255b7aa070b2a70acb90803864a40f05d0ec47556ddd0" already present on machine 2026/07/08 19:01:20 19:00:00 INFO Job/maas-api-key-cleanup-29725620: SuccessfulCreate - Created pod: maas-api-key-cleanup-29725620-spctg 2026/07/08 19:01:20 19:00:00 INFO CronJob/maas-api-key-cleanup: SuccessfulCreate - Created job maas-api-key-cleanup-29725620 2026/07/08 19:01:20 18:59:53 WARN Pod/maas-controller-656b6867cb-x6k8q: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-qsmdx); attempting to pull the image may not succeed. 2026/07/08 19:01:20 === RESOURCE QUOTAS === 2026/07/08 19:01:20 No resource quotas found in monitored namespaces 2026/07/08 19:01:20 Classification for DSCInitialization and DataScienceCluster validation E2E Tests: infrastructure/dsci-unhealthy (code=1009, confidence=medium) 2026/07/08 19:01:20 Evidence: DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"dsci-unhealthy","error_code":1009,"evidence":["DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False"],"confidence":"medium"} 2026/07/08 19:01:20 Diagnostics complete for failed test: DSCInitialization and DataScienceCluster validation E2E Tests controller_test.go:478: Stopping: DSCInitialization and DataScienceCluster validation E2E Tests test failed. 2026/07/08 19:01:20 Circuit breaker: failure classified as infrastructure/dsci-unhealthy (code=1009) 2026/07/08 19:01:20 Circuit breaker: infrastructure/unknown failure recorded (1/3) 2026/07/08 19:26:58 TEST FAILURE DETECTED: components 2026/07/08 19:26:58 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/08 19:26:58 === CLUSTER STATE === 2026/07/08 19:26:58 Node: ods-qe-psi-07-b4mqp-master-0 2026/07/08 19:26:58 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:26:58 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:26:58 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:26:58 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:26:58 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/08 19:26:58 Node: ods-qe-psi-07-b4mqp-master-1 2026/07/08 19:26:58 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:26:58 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:26:58 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:26:58 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:26:58 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/08 19:26:58 Node: ods-qe-psi-07-b4mqp-master-2 2026/07/08 19:26:58 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:26:58 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:26:58 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:26:58 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:26:58 Resources: 11500m CPU, 23450732Ki memory allocatable (of 12000m CPU, 24601708Ki memory capacity) 2026/07/08 19:26:58 Node: ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/08 19:26:58 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:26:58 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:26:58 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:26:58 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:26:58 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/08 19:26:58 Node: ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 19:26:58 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:26:58 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:26:58 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:26:58 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:26:58 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/08 19:26:58 Node: ods-qe-psi-07-b4mqp-worker-0-jd4pm 2026/07/08 19:26:58 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:26:58 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:26:58 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:26:58 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:26:58 Resources: 11500m CPU, 23452736Ki memory allocatable (of 12000m CPU, 24603712Ki memory capacity) 2026/07/08 19:26:58 Node: ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/08 19:26:58 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:26:58 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:26:58 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:26:58 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:26:58 Resources: 11500m CPU, 23452744Ki memory allocatable (of 12000m CPU, 24603720Ki memory capacity) 2026/07/08 19:26:58 Node: ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/08 19:26:58 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/08 19:26:58 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/08 19:26:58 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/08 19:26:58 Condition Ready: True - kubelet is posting ready status 2026/07/08 19:26:58 Resources: 11500m CPU, 23450720Ki memory allocatable (of 12000m CPU, 24601696Ki memory capacity) 2026/07/08 19:26:58 === DEPLOYMENTS === 2026/07/08 19:26:58 Namespace: redhat-ods-applications 2026/07/08 19:26:58 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/08 19:26:58 Deployment kubeflow-trainer-controller-manager: 1/1 (READY) 2026/07/08 19:26:58 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/08 19:26:58 Deployment maas-postgres: 1/1 (READY) 2026/07/08 19:26:58 Deployment model-serving-api: 1/1 (READY) 2026/07/08 19:26:58 Deployment odh-model-controller: 1/1 (READY) 2026/07/08 19:26:58 Deployment postgres: 1/1 (READY) 2026/07/08 19:26:58 Namespace: kube-system 2026/07/08 19:26:58 === PODS === 2026/07/08 19:26:58 Namespace redhat-ods-applications: no problematic pods found 2026/07/08 19:26:58 Namespace kube-system: no problematic pods found 2026/07/08 19:26:58 === OPERATOR STATUS === 2026/07/08 19:26:58 Operator deployment: 3/3 ready 2026/07/08 19:26:58 === DSCI STATUS === 2026/07/08 19:26:58 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/08 19:26:58 === DSC STATUS === 2026/07/08 19:26:58 DSC default-dsc: 2026/07/08 19:26:58 Ready: True - 2026/07/08 19:26:58 ProvisioningSucceeded: True - 2026/07/08 19:26:58 ComponentsReady: True - 2026/07/08 19:26:58 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/08 19:26:58 DashboardReady: False - Component ManagementState is set to Removed 2026/07/08 19:26:58 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/08 19:26:58 KserveLLMInferenceServiceDependencies: True - 2026/07/08 19:26:58 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/08 19:26:58 KserveReady: True - 2026/07/08 19:26:58 KueueReady: True - 2026/07/08 19:26:58 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/08 19:26:58 ModelControllerReady: True - 2026/07/08 19:26:58 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/08 19:26:58 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/08 19:26:58 OGXReady: False - Component ManagementState is set to Removed 2026/07/08 19:26:58 RayReady: False - Component ManagementState is set to Removed 2026/07/08 19:26:58 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/08 19:26:58 TrainerReady: True - 2026/07/08 19:26:58 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/08 19:26:58 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/08 19:26:58 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/08 19:26:58 === RECENT EVENTS === 2026/07/08 19:26:58 19:26:32 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-586c85c465-d7vnt_c501cb0e-5517-42e1-95ac-7599a9539b77 became leader 2026/07/08 19:26:58 19:26:30 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c8cb86d7f-st4s8_a6305d9a-7aed-47d2-b691-bce02a0763fe became leader 2026/07/08 19:26:58 19:26:17 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-54bf759f64-7dcjv_d7671f0c-3c6c-42fe-93d2-a1aa9a900882 became leader 2026/07/08 19:26:58 19:26:15 INFO Pod/llmisvc-controller-manager-586c85c465-d7vnt: Created - Created container: manager 2026/07/08 19:26:58 19:26:15 INFO Pod/llmisvc-controller-manager-586c85c465-d7vnt: Started - Started container manager 2026/07/08 19:26:58 19:26:15 INFO Pod/kserve-controller-manager-6c8cb86d7f-st4s8: Started - Started container manager 2026/07/08 19:26:58 19:26:15 INFO Pod/llmisvc-controller-manager-586c85c465-d7vnt: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:d3225eb3f0bfcb20a6d1dccd63403e58a810921680425ecae1b22d86a451417d" in 854ms (854ms including waiting). Image size: 235185383 bytes. 2026/07/08 19:26:58 19:26:15 INFO Pod/kserve-controller-manager-6c8cb86d7f-st4s8: Created - Created container: manager 2026/07/08 19:26:58 19:26:15 INFO Pod/kserve-controller-manager-6c8cb86d7f-st4s8: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:7f6f393635a7431d530256ef8ad48ebd35531166dbb361c6c0ff9fc37bd5dc2b" in 770ms (770ms including waiting). Image size: 237680379 bytes. 2026/07/08 19:26:58 19:26:14 INFO Pod/kserve-controller-manager-6c8cb86d7f-st4s8: AddedInterface - Add eth0 [10.129.3.45/23] from ovn-kubernetes 2026/07/08 19:26:58 19:26:14 INFO Pod/kserve-controller-manager-6c8cb86d7f-st4s8: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:7f6f393635a7431d530256ef8ad48ebd35531166dbb361c6c0ff9fc37bd5dc2b" 2026/07/08 19:26:58 19:26:14 INFO Pod/llmisvc-controller-manager-586c85c465-d7vnt: AddedInterface - Add eth0 [10.129.3.46/23] from ovn-kubernetes 2026/07/08 19:26:58 19:26:14 INFO Pod/llmisvc-controller-manager-586c85c465-d7vnt: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:d3225eb3f0bfcb20a6d1dccd63403e58a810921680425ecae1b22d86a451417d" 2026/07/08 19:26:58 19:26:13 INFO Pod/llmisvc-controller-manager-586c85c465-d7vnt: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-586c85c465-d7vnt to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 19:26:58 19:26:13 INFO Pod/kserve-controller-manager-6c8cb86d7f-st4s8: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c8cb86d7f-st4s8 to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 19:26:58 19:26:13 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-586c85c465 from 0 to 1 2026/07/08 19:26:58 19:26:13 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c8cb86d7f from 0 to 1 2026/07/08 19:26:58 19:26:13 INFO ReplicaSet/llmisvc-controller-manager-586c85c465: SuccessfulCreate - Created pod: llmisvc-controller-manager-586c85c465-d7vnt 2026/07/08 19:26:58 19:26:13 INFO ReplicaSet/kserve-controller-manager-6c8cb86d7f: SuccessfulCreate - Created pod: kserve-controller-manager-6c8cb86d7f-st4s8 2026/07/08 19:26:58 19:26:00 INFO Pod/model-serving-api-78bbbb9bf8-zmqbh: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-78bbbb9bf8-zmqbh to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/08 19:26:58 19:26:00 INFO Pod/model-serving-api-78bbbb9bf8-zmqbh: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:6dd699e678c3f4d6e51138eea62f096f26472b5f9c1ce2757cf9e9e2f80371a4" already present on machine 2026/07/08 19:26:58 19:26:00 INFO Pod/odh-model-controller-54bf759f64-7dcjv: Created - Created container: manager 2026/07/08 19:26:58 19:26:00 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-78bbbb9bf8 from 0 to 1 2026/07/08 19:26:58 19:26:00 INFO Pod/odh-model-controller-54bf759f64-7dcjv: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b423702816000ce0f075548e3a67de006f40ccf05c575a866cb62e2ac9711191" in 745ms (745ms including waiting). Image size: 238719956 bytes. 2026/07/08 19:26:58 19:26:00 INFO ReplicaSet/model-serving-api-78bbbb9bf8: SuccessfulCreate - Created pod: model-serving-api-78bbbb9bf8-zmqbh 2026/07/08 19:26:58 19:26:00 INFO Pod/model-serving-api-78bbbb9bf8-zmqbh: Started - Started container server 2026/07/08 19:26:58 19:26:00 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/08 19:26:58 19:26:00 INFO Pod/model-serving-api-78bbbb9bf8-zmqbh: Created - Created container: server 2026/07/08 19:26:58 19:26:00 WARN ReplicaSet/model-serving-api-78bbbb9bf8: FailedCreate - Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/08 19:26:58 19:26:00 INFO Pod/odh-model-controller-54bf759f64-7dcjv: Started - Started container manager 2026/07/08 19:26:58 19:26:00 INFO Pod/model-serving-api-78bbbb9bf8-zmqbh: AddedInterface - Add eth0 [10.130.3.13/23] from ovn-kubernetes 2026/07/08 19:26:58 19:25:59 INFO Pod/odh-model-controller-54bf759f64-7dcjv: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b423702816000ce0f075548e3a67de006f40ccf05c575a866cb62e2ac9711191" 2026/07/08 19:26:58 19:25:59 INFO Pod/odh-model-controller-54bf759f64-7dcjv: AddedInterface - Add eth0 [10.129.3.44/23] from ovn-kubernetes 2026/07/08 19:26:58 19:25:58 INFO Pod/odh-model-controller-54bf759f64-7dcjv: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-54bf759f64-7dcjv to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 19:26:58 19:25:58 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-54bf759f64 from 0 to 1 2026/07/08 19:26:58 19:25:58 INFO ReplicaSet/odh-model-controller-54bf759f64: SuccessfulCreate - Created pod: odh-model-controller-54bf759f64-7dcjv 2026/07/08 19:26:58 19:25:48 INFO Pod/model-serving-api-78bbbb9bf8-xprbp: Killing - Stopping container server 2026/07/08 19:26:58 19:25:48 INFO Pod/kserve-controller-manager-6c8cb86d7f-r5bq4: Killing - Stopping container manager 2026/07/08 19:26:58 19:25:48 INFO Pod/llmisvc-controller-manager-586c85c465-5kbwl: Killing - Stopping container manager 2026/07/08 19:26:58 19:25:48 INFO Pod/odh-model-controller-54bf759f64-66j4r: Killing - Stopping container manager 2026/07/08 19:26:58 19:25:47 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/08 19:26:58 19:25:22 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-586c85c465-5kbwl_4026e0bb-ab43-44dd-a3a2-c61403dc1275 became leader 2026/07/08 19:26:58 19:25:19 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c8cb86d7f-r5bq4_1463b9eb-c303-4dec-b2f9-054cd116771e became leader 2026/07/08 19:26:58 19:25:06 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-54bf759f64-66j4r_1000815e-2e03-48d1-94b0-cf9fe4bab628 became leader 2026/07/08 19:26:58 19:25:04 INFO Pod/kserve-controller-manager-6c8cb86d7f-r5bq4: Created - Created container: manager 2026/07/08 19:26:58 19:25:04 INFO Pod/llmisvc-controller-manager-586c85c465-5kbwl: Created - Created container: manager 2026/07/08 19:26:58 19:25:04 INFO Pod/llmisvc-controller-manager-586c85c465-5kbwl: Started - Started container manager 2026/07/08 19:26:58 19:25:04 INFO Pod/kserve-controller-manager-6c8cb86d7f-r5bq4: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:7f6f393635a7431d530256ef8ad48ebd35531166dbb361c6c0ff9fc37bd5dc2b" in 706ms (706ms including waiting). Image size: 237680379 bytes. 2026/07/08 19:26:58 19:25:04 INFO Pod/kserve-controller-manager-6c8cb86d7f-r5bq4: Started - Started container manager 2026/07/08 19:26:58 19:25:04 INFO Pod/llmisvc-controller-manager-586c85c465-5kbwl: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:d3225eb3f0bfcb20a6d1dccd63403e58a810921680425ecae1b22d86a451417d" in 826ms (826ms including waiting). Image size: 235185383 bytes. 2026/07/08 19:26:58 19:25:03 INFO Pod/llmisvc-controller-manager-586c85c465-5kbwl: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-586c85c465-5kbwl to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 19:26:58 19:25:03 INFO Pod/kserve-controller-manager-6c8cb86d7f-r5bq4: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c8cb86d7f-r5bq4 to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 19:26:58 19:25:03 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c8cb86d7f from 0 to 1 2026/07/08 19:26:58 19:25:03 INFO ReplicaSet/llmisvc-controller-manager-586c85c465: SuccessfulCreate - Created pod: llmisvc-controller-manager-586c85c465-5kbwl 2026/07/08 19:26:58 19:25:03 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-586c85c465 from 0 to 1 2026/07/08 19:26:58 19:25:03 INFO Pod/llmisvc-controller-manager-586c85c465-5kbwl: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:d3225eb3f0bfcb20a6d1dccd63403e58a810921680425ecae1b22d86a451417d" 2026/07/08 19:26:58 19:25:03 INFO Pod/llmisvc-controller-manager-586c85c465-5kbwl: AddedInterface - Add eth0 [10.129.3.43/23] from ovn-kubernetes 2026/07/08 19:26:58 19:25:03 INFO ReplicaSet/kserve-controller-manager-6c8cb86d7f: SuccessfulCreate - Created pod: kserve-controller-manager-6c8cb86d7f-r5bq4 2026/07/08 19:26:58 19:25:03 INFO Pod/kserve-controller-manager-6c8cb86d7f-r5bq4: AddedInterface - Add eth0 [10.129.3.42/23] from ovn-kubernetes 2026/07/08 19:26:58 19:25:03 INFO Pod/kserve-controller-manager-6c8cb86d7f-r5bq4: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:7f6f393635a7431d530256ef8ad48ebd35531166dbb361c6c0ff9fc37bd5dc2b" 2026/07/08 19:26:58 19:25:00 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-776d978f97-9mdsl_82a22f8e-71af-4851-97c5-f807698e6d50 became leader 2026/07/08 19:26:58 19:24:49 INFO Pod/model-serving-api-78bbbb9bf8-xprbp: Created - Created container: server 2026/07/08 19:26:58 19:24:49 INFO Pod/odh-model-controller-54bf759f64-66j4r: Created - Created container: manager 2026/07/08 19:26:58 19:24:49 INFO Pod/model-serving-api-78bbbb9bf8-xprbp: Started - Started container server 2026/07/08 19:26:58 19:24:49 INFO Pod/odh-model-controller-54bf759f64-66j4r: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b423702816000ce0f075548e3a67de006f40ccf05c575a866cb62e2ac9711191" in 1.432s (1.432s including waiting). Image size: 238719956 bytes. 2026/07/08 19:26:58 19:24:49 INFO Pod/odh-model-controller-54bf759f64-66j4r: Started - Started container manager 2026/07/08 19:26:58 19:24:49 INFO Pod/model-serving-api-78bbbb9bf8-xprbp: AddedInterface - Add eth0 [10.130.3.12/23] from ovn-kubernetes 2026/07/08 19:26:58 19:24:49 INFO Pod/model-serving-api-78bbbb9bf8-xprbp: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:6dd699e678c3f4d6e51138eea62f096f26472b5f9c1ce2757cf9e9e2f80371a4" already present on machine 2026/07/08 19:26:58 19:24:48 INFO Pod/model-serving-api-78bbbb9bf8-xprbp: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-78bbbb9bf8-xprbp to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/08 19:26:58 19:24:48 INFO Pod/odh-model-controller-54bf759f64-66j4r: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b423702816000ce0f075548e3a67de006f40ccf05c575a866cb62e2ac9711191" 2026/07/08 19:26:58 19:24:48 INFO Pod/odh-model-controller-54bf759f64-66j4r: AddedInterface - Add eth0 [10.129.3.41/23] from ovn-kubernetes 2026/07/08 19:26:58 19:24:48 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-78bbbb9bf8 from 0 to 1 2026/07/08 19:26:58 19:24:48 WARN ReplicaSet/model-serving-api-78bbbb9bf8: FailedCreate - Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/08 19:26:58 19:24:48 INFO ReplicaSet/model-serving-api-78bbbb9bf8: SuccessfulCreate - Created pod: model-serving-api-78bbbb9bf8-xprbp 2026/07/08 19:26:58 19:24:47 INFO Pod/odh-model-controller-54bf759f64-66j4r: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-54bf759f64-66j4r to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 19:26:58 19:24:47 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-54bf759f64 from 0 to 1 2026/07/08 19:26:58 19:24:47 INFO ReplicaSet/odh-model-controller-54bf759f64: SuccessfulCreate - Created pod: odh-model-controller-54bf759f64-66j4r 2026/07/08 19:26:58 19:24:47 WARN Pod/odh-model-controller-54bf759f64-66j4r: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/08 19:26:58 19:24:46 INFO Pod/kserve-controller-manager-6c8cb86d7f-z2t7f: Created - Created container: manager 2026/07/08 19:26:58 19:24:46 INFO Pod/llmisvc-controller-manager-586c85c465-pdmf5: Started - Started container manager 2026/07/08 19:26:58 19:24:46 INFO Pod/llmisvc-controller-manager-586c85c465-pdmf5: Created - Created container: manager 2026/07/08 19:26:58 19:24:46 INFO Pod/llmisvc-controller-manager-586c85c465-pdmf5: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:d3225eb3f0bfcb20a6d1dccd63403e58a810921680425ecae1b22d86a451417d" in 757ms (757ms including waiting). Image size: 235185383 bytes. 2026/07/08 19:26:58 19:24:46 INFO Pod/kserve-controller-manager-6c8cb86d7f-z2t7f: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:7f6f393635a7431d530256ef8ad48ebd35531166dbb361c6c0ff9fc37bd5dc2b" in 822ms (822ms including waiting). Image size: 237680379 bytes. 2026/07/08 19:26:58 19:24:46 INFO Pod/kserve-controller-manager-6c8cb86d7f-z2t7f: Started - Started container manager 2026/07/08 19:26:58 19:24:46 INFO Pod/kserve-controller-manager-6c8cb86d7f-z2t7f: Killing - Stopping container manager 2026/07/08 19:26:58 19:24:45 INFO Pod/llmisvc-controller-manager-586c85c465-pdmf5: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-586c85c465-pdmf5 to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 19:26:58 19:24:45 INFO Pod/kserve-controller-manager-6c8cb86d7f-z2t7f: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c8cb86d7f-z2t7f to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 19:26:58 19:24:45 INFO ReplicaSet/kserve-controller-manager-6c8cb86d7f: SuccessfulCreate - Created pod: kserve-controller-manager-6c8cb86d7f-z2t7f 2026/07/08 19:26:58 19:24:45 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-586c85c465 from 0 to 1 2026/07/08 19:26:58 19:24:45 INFO Pod/llmisvc-controller-manager-586c85c465-pdmf5: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:d3225eb3f0bfcb20a6d1dccd63403e58a810921680425ecae1b22d86a451417d" 2026/07/08 19:26:58 19:24:45 INFO Pod/kserve-controller-manager-6c8cb86d7f-z2t7f: AddedInterface - Add eth0 [10.129.3.39/23] from ovn-kubernetes 2026/07/08 19:26:58 19:24:45 INFO Pod/kserve-controller-manager-6c8cb86d7f-z2t7f: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:7f6f393635a7431d530256ef8ad48ebd35531166dbb361c6c0ff9fc37bd5dc2b" 2026/07/08 19:26:58 19:24:45 INFO Pod/llmisvc-controller-manager-586c85c465-pdmf5: AddedInterface - Add eth0 [10.129.3.40/23] from ovn-kubernetes 2026/07/08 19:26:58 19:24:45 WARN Pod/llmisvc-controller-manager-586c85c465-pdmf5: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (llmisvc-controller-manager-dockercfg-d5g2m); attempting to pull the image may not succeed. 2026/07/08 19:26:58 19:24:45 INFO ReplicaSet/llmisvc-controller-manager-586c85c465: SuccessfulCreate - Created pod: llmisvc-controller-manager-586c85c465-pdmf5 2026/07/08 19:26:58 19:24:44 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c8cb86d7f from 0 to 1 2026/07/08 19:26:58 19:24:43 INFO Pod/kubeflow-trainer-controller-manager-776d978f97-9mdsl: Started - Started container manager 2026/07/08 19:26:58 19:24:43 INFO Pod/kubeflow-trainer-controller-manager-776d978f97-9mdsl: AddedInterface - Add eth0 [10.129.3.38/23] from ovn-kubernetes 2026/07/08 19:26:58 19:24:43 INFO Pod/kubeflow-trainer-controller-manager-776d978f97-9mdsl: Pulled - Container image "registry.redhat.io/rhoai/odh-trainer-rhel9@sha256:904dd86da976fced2d3618cbe932baadaf94934280aab3a68f6cc9f04ee2efbd" already present on machine 2026/07/08 19:26:58 19:24:43 INFO Pod/kubeflow-trainer-controller-manager-776d978f97-9mdsl: Created - Created container: manager 2026/07/08 19:26:58 19:24:42 INFO Pod/kubeflow-trainer-controller-manager-776d978f97-9mdsl: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-776d978f97-9mdsl to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 19:26:58 19:24:42 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-776d978f97 from 0 to 1 2026/07/08 19:26:58 19:24:42 INFO ReplicaSet/kubeflow-trainer-controller-manager-776d978f97: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-776d978f97-9mdsl 2026/07/08 19:26:58 19:24:38 INFO Pod/model-serving-api-78bbbb9bf8-sqnr2: Killing - Stopping container server 2026/07/08 19:26:58 19:24:38 WARN Service/llmisvc-webhook-server-service: FailedToUpdateEndpointSlices - Error updating Endpoint Slices for Service redhat-ods-applications/llmisvc-webhook-server-service: failed to update llmisvc-webhook-server-service-nlbpz EndpointSlice for Service redhat-ods-applications/llmisvc-webhook-server-service: Operation cannot be fulfilled on endpointslices.discovery.k8s.io "llmisvc-webhook-server-service-nlbpz": StorageError: invalid object, Code: 4, Key: /kubernetes.io/endpointslices/redhat-ods-applications/llmisvc-webhook-server-service-nlbpz, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: e475e32a-d21b-42e8-9912-33edf89d91d2, UID in object meta: 2026/07/08 19:26:58 19:24:38 WARN Service/llmisvc-controller-manager-service: FailedToUpdateEndpointSlices - Error updating Endpoint Slices for Service redhat-ods-applications/llmisvc-controller-manager-service: failed to update llmisvc-controller-manager-service-wb5q6 EndpointSlice for Service redhat-ods-applications/llmisvc-controller-manager-service: Operation cannot be fulfilled on endpointslices.discovery.k8s.io "llmisvc-controller-manager-service-wb5q6": StorageError: invalid object, Code: 4, Key: /kubernetes.io/endpointslices/redhat-ods-applications/llmisvc-controller-manager-service-wb5q6, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: b695f0d2-4a17-4e7f-af24-08920c097913, UID in object meta: 2026/07/08 19:26:58 19:24:38 INFO Pod/llmisvc-controller-manager-586c85c465-bpfs4: Killing - Stopping container manager 2026/07/08 19:26:58 19:24:38 INFO Pod/kserve-controller-manager-6c8cb86d7f-pbtgj: Killing - Stopping container manager 2026/07/08 19:26:58 19:24:38 INFO Pod/odh-model-controller-54bf759f64-6xs54: Killing - Stopping container manager 2026/07/08 19:26:58 19:24:26 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/08 19:26:58 19:24:26 INFO Pod/maas-controller-656b6867cb-9ntbf: Killing - Stopping container manager 2026/07/08 19:26:58 19:24:26 INFO Pod/maas-api-84c4b8dcb6-lf594: Killing - Stopping container maas-api 2026/07/08 19:26:58 19:24:15 INFO Lease/maas-controller.models-as-a-service.opendatahub.io: LeaderElection - maas-controller-656b6867cb-9ntbf_253c3ab8-be66-4940-bf03-8add980ee0ba became leader 2026/07/08 19:26:58 19:23:58 INFO Pod/maas-api-84c4b8dcb6-lf594: Started - Started container maas-api 2026/07/08 19:26:58 19:23:58 INFO Pod/maas-controller-656b6867cb-9ntbf: Created - Created container: manager 2026/07/08 19:26:58 19:23:58 INFO Pod/maas-controller-656b6867cb-9ntbf: Started - Started container manager 2026/07/08 19:26:58 19:23:58 INFO Pod/maas-controller-656b6867cb-9ntbf: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:b8cca0d56d9ca5ab366cdc4042f13a908326e57b1b711c59e6ff101507058fb4" in 697ms (697ms including waiting). Image size: 212345923 bytes. 2026/07/08 19:26:58 19:23:58 INFO Pod/maas-api-84c4b8dcb6-lf594: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-api-rhel9@sha256:f1cc3f00d35421d0b60782cdb4e0331d94e922b483bc1f0e837ceabe63570328" in 842ms (842ms including waiting). Image size: 174987716 bytes. 2026/07/08 19:26:58 19:23:58 INFO Pod/maas-api-84c4b8dcb6-lf594: Created - Created container: maas-api 2026/07/08 19:26:58 19:23:57 INFO Pod/maas-api-84c4b8dcb6-lf594: AddedInterface - Add eth0 [10.130.3.10/23] from ovn-kubernetes 2026/07/08 19:26:58 19:23:57 INFO Pod/maas-api-84c4b8dcb6-lf594: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-api-rhel9@sha256:f1cc3f00d35421d0b60782cdb4e0331d94e922b483bc1f0e837ceabe63570328" 2026/07/08 19:26:58 19:23:57 INFO Pod/maas-controller-656b6867cb-9ntbf: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:b8cca0d56d9ca5ab366cdc4042f13a908326e57b1b711c59e6ff101507058fb4" 2026/07/08 19:26:58 19:23:56 INFO Pod/maas-api-84c4b8dcb6-lf594: Scheduled - Successfully assigned redhat-ods-applications/maas-api-84c4b8dcb6-lf594 to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/08 19:26:58 19:23:56 INFO ReplicaSet/maas-api-84c4b8dcb6: SuccessfulCreate - Created pod: maas-api-84c4b8dcb6-lf594 2026/07/08 19:26:58 19:23:56 WARN ReplicaSet/maas-api-84c4b8dcb6: FailedCreate - Error creating: pods "maas-api-84c4b8dcb6-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 2026/07/08 19:26:58 19:23:56 INFO Deployment/maas-api: ScalingReplicaSet - Scaled up replica set maas-api-84c4b8dcb6 from 0 to 1 2026/07/08 19:26:58 19:23:45 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-586c85c465-bpfs4_7faf6b47-a5f6-46a5-8996-3698fd29c2ab became leader 2026/07/08 19:26:58 19:23:43 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c8cb86d7f-pbtgj_9eef8101-a7c4-4205-9bb1-65bf7c1c758b became leader 2026/07/08 19:26:58 19:23:28 INFO Pod/kserve-controller-manager-6c8cb86d7f-pbtgj: Started - Started container manager 2026/07/08 19:26:58 19:23:28 INFO Pod/kserve-controller-manager-6c8cb86d7f-pbtgj: Created - Created container: manager 2026/07/08 19:26:58 19:23:28 INFO Pod/llmisvc-controller-manager-586c85c465-bpfs4: Created - Created container: manager 2026/07/08 19:26:58 19:23:28 INFO Pod/llmisvc-controller-manager-586c85c465-bpfs4: Started - Started container manager 2026/07/08 19:26:58 19:23:28 INFO Pod/llmisvc-controller-manager-586c85c465-bpfs4: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:d3225eb3f0bfcb20a6d1dccd63403e58a810921680425ecae1b22d86a451417d" in 1.089s (1.089s including waiting). Image size: 235185383 bytes. 2026/07/08 19:26:58 19:23:28 INFO Pod/kserve-controller-manager-6c8cb86d7f-pbtgj: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:7f6f393635a7431d530256ef8ad48ebd35531166dbb361c6c0ff9fc37bd5dc2b" in 697ms (697ms including waiting). Image size: 237680379 bytes. 2026/07/08 19:26:58 19:23:27 INFO Pod/llmisvc-controller-manager-586c85c465-bpfs4: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:d3225eb3f0bfcb20a6d1dccd63403e58a810921680425ecae1b22d86a451417d" 2026/07/08 19:26:58 19:23:27 INFO Pod/kserve-controller-manager-6c8cb86d7f-pbtgj: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:7f6f393635a7431d530256ef8ad48ebd35531166dbb361c6c0ff9fc37bd5dc2b" 2026/07/08 19:26:58 19:23:27 INFO Pod/kserve-controller-manager-6c8cb86d7f-pbtgj: AddedInterface - Add eth0 [10.129.3.36/23] from ovn-kubernetes 2026/07/08 19:26:58 19:23:27 INFO Pod/llmisvc-controller-manager-586c85c465-bpfs4: AddedInterface - Add eth0 [10.129.3.37/23] from ovn-kubernetes 2026/07/08 19:26:58 19:23:26 INFO Pod/llmisvc-controller-manager-586c85c465-bpfs4: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-586c85c465-bpfs4 to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 19:26:58 19:23:26 INFO Pod/kserve-controller-manager-6c8cb86d7f-pbtgj: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c8cb86d7f-pbtgj to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 19:26:58 19:23:26 INFO ReplicaSet/kserve-controller-manager-6c8cb86d7f: SuccessfulCreate - Created pod: kserve-controller-manager-6c8cb86d7f-pbtgj 2026/07/08 19:26:58 19:23:26 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-54bf759f64-6xs54_f26ff129-f881-4870-a205-adc222831225 became leader 2026/07/08 19:26:58 19:23:26 INFO ReplicaSet/llmisvc-controller-manager-586c85c465: SuccessfulCreate - Created pod: llmisvc-controller-manager-586c85c465-bpfs4 2026/07/08 19:26:58 19:23:26 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-586c85c465 from 0 to 1 2026/07/08 19:26:58 19:23:26 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c8cb86d7f from 0 to 1 2026/07/08 19:26:58 19:23:23 INFO Lease/maas-controller.models-as-a-service.opendatahub.io: LeaderElection - maas-controller-656b6867cb-9ntbf_5035efe7-3476-46fc-a2e1-0c115d30fd07 became leader 2026/07/08 19:26:58 19:23:09 INFO Pod/model-serving-api-78bbbb9bf8-sqnr2: Created - Created container: server 2026/07/08 19:26:58 19:23:09 INFO Pod/model-serving-api-78bbbb9bf8-sqnr2: Started - Started container server 2026/07/08 19:26:58 19:23:09 INFO Pod/model-serving-api-78bbbb9bf8-sqnr2: AddedInterface - Add eth0 [10.129.3.35/23] from ovn-kubernetes 2026/07/08 19:26:58 19:23:09 INFO Pod/model-serving-api-78bbbb9bf8-sqnr2: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:6dd699e678c3f4d6e51138eea62f096f26472b5f9c1ce2757cf9e9e2f80371a4" already present on machine 2026/07/08 19:26:58 19:23:08 INFO Pod/model-serving-api-78bbbb9bf8-sqnr2: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-78bbbb9bf8-sqnr2 to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 19:26:58 19:23:08 INFO Pod/maas-controller-656b6867cb-9ntbf: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:b8cca0d56d9ca5ab366cdc4042f13a908326e57b1b711c59e6ff101507058fb4" in 719ms (719ms including waiting). Image size: 212345923 bytes. 2026/07/08 19:26:58 19:23:08 INFO Pod/odh-model-controller-54bf759f64-6xs54: Started - Started container manager 2026/07/08 19:26:58 19:23:08 INFO Pod/odh-model-controller-54bf759f64-6xs54: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b423702816000ce0f075548e3a67de006f40ccf05c575a866cb62e2ac9711191" in 766ms (766ms including waiting). Image size: 238719956 bytes. 2026/07/08 19:26:58 19:23:08 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-78bbbb9bf8 from 0 to 1 2026/07/08 19:26:58 19:23:08 WARN ReplicaSet/model-serving-api-78bbbb9bf8: FailedCreate - Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/08 19:26:58 19:23:08 INFO ReplicaSet/model-serving-api-78bbbb9bf8: SuccessfulCreate - Created pod: model-serving-api-78bbbb9bf8-sqnr2 2026/07/08 19:26:58 19:23:08 INFO Pod/odh-model-controller-54bf759f64-6xs54: Created - Created container: manager 2026/07/08 19:26:58 19:23:07 INFO Pod/maas-controller-656b6867cb-9ntbf: Scheduled - Successfully assigned redhat-ods-applications/maas-controller-656b6867cb-9ntbf to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 19:26:58 19:23:07 INFO Pod/odh-model-controller-54bf759f64-6xs54: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b423702816000ce0f075548e3a67de006f40ccf05c575a866cb62e2ac9711191" 2026/07/08 19:26:58 19:23:07 INFO Pod/odh-model-controller-54bf759f64-6xs54: AddedInterface - Add eth0 [10.129.3.33/23] from ovn-kubernetes 2026/07/08 19:26:58 19:23:07 INFO ReplicaSet/maas-controller-656b6867cb: SuccessfulCreate - Created pod: maas-controller-656b6867cb-9ntbf 2026/07/08 19:26:58 19:23:07 INFO Deployment/maas-controller: ScalingReplicaSet - Scaled up replica set maas-controller-656b6867cb from 0 to 1 2026/07/08 19:26:58 19:23:07 INFO Pod/maas-controller-656b6867cb-9ntbf: AddedInterface - Add eth0 [10.129.3.34/23] from ovn-kubernetes 2026/07/08 19:26:58 19:23:06 INFO Pod/odh-model-controller-54bf759f64-6xs54: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-54bf759f64-6xs54 to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 19:26:58 19:23:06 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-54bf759f64 from 0 to 1 2026/07/08 19:26:58 19:23:06 WARN Pod/odh-model-controller-54bf759f64-6xs54: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/08 19:26:58 19:23:06 INFO ReplicaSet/odh-model-controller-54bf759f64: SuccessfulCreate - Created pod: odh-model-controller-54bf759f64-6xs54 2026/07/08 19:26:58 19:22:55 INFO Pod/trustyai-service-operator-controller-manager-6f7f88c874-rm774: Killing - Stopping container manager 2026/07/08 19:26:58 19:22:45 INFO Pod/llmisvc-controller-manager-586c85c465-94ns9: Killing - Stopping container manager 2026/07/08 19:26:58 19:22:45 INFO Pod/model-serving-api-78bbbb9bf8-9qztk: Killing - Stopping container server 2026/07/08 19:26:58 19:22:45 INFO Pod/kserve-controller-manager-6c8cb86d7f-wgbf2: Killing - Stopping container manager 2026/07/08 19:26:58 19:22:45 INFO Pod/maas-controller-656b6867cb-4zthx: Killing - Stopping container manager 2026/07/08 19:26:58 19:22:45 INFO Pod/maas-api-84c4b8dcb6-sfzzd: Killing - Stopping container maas-api 2026/07/08 19:26:58 19:22:45 INFO Lease/b7e9931f.trustyai.opendatahub.io: LeaderElection - trustyai-service-operator-controller-manager-6f7f88c874-rm774_5860bea9-6054-4946-9c20-33d1cf6689d1 became leader 2026/07/08 19:26:58 19:22:45 INFO Pod/odh-model-controller-54bf759f64-tfb5n: Killing - Stopping container manager 2026/07/08 19:26:58 19:22:30 INFO Pod/trustyai-service-operator-controller-manager-6f7f88c874-rm774: AddedInterface - Add eth0 [10.129.3.32/23] from ovn-kubernetes 2026/07/08 19:26:58 19:22:30 INFO Pod/trustyai-service-operator-controller-manager-6f7f88c874-rm774: Pulled - Container image "registry.redhat.io/rhoai/odh-trustyai-service-operator-rhel9@sha256:b9d442a66d0ac181c430fd5f27b6de0dac8b4f815ca9668c436da246676f0980" already present on machine 2026/07/08 19:26:58 19:22:30 INFO Pod/trustyai-service-operator-controller-manager-6f7f88c874-rm774: Created - Created container: manager 2026/07/08 19:26:58 19:22:30 INFO Pod/trustyai-service-operator-controller-manager-6f7f88c874-rm774: Started - Started container manager 2026/07/08 19:26:58 19:22:29 INFO Pod/trustyai-service-operator-controller-manager-6f7f88c874-rm774: Scheduled - Successfully assigned redhat-ods-applications/trustyai-service-operator-controller-manager-6f7f88c874-rm774 to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/08 19:26:58 19:22:29 INFO Pod/trustyai-service-operator-controller-manager-6f7f88c874-hg5wv: Killing - Stopping container manager 2026/07/08 19:26:58 19:22:29 INFO Pod/trustyai-service-operator-controller-manager-6f7f88c874-w7cgw: Killing - Stopping container manager 2026/07/08 19:26:58 19:22:29 INFO ReplicaSet/trustyai-service-operator-controller-manager-6f7f88c874: SuccessfulCreate - Created pod: trustyai-service-operator-controller-manager-6f7f88c874-rm774 2026/07/08 19:26:58 19:22:29 INFO Deployment/trustyai-service-operator-controller-manager: ScalingReplicaSet - Scaled up replica set trustyai-service-operator-controller-manager-6f7f88c874 from 0 to 1 2026/07/08 19:26:58 19:22:00 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/08 19:26:58 === RESOURCE QUOTAS === 2026/07/08 19:26:58 No resource quotas found in monitored namespaces 2026/07/08 19:26:58 Classification for components: infrastructure/rbac (code=1011, confidence=medium) 2026/07/08 19:26:58 Evidence: event ReplicaSet/model-serving-api-78bbbb9bf8: FailedCreate - Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"rbac","error_code":1011,"evidence":["event ReplicaSet/model-serving-api-78bbbb9bf8: FailedCreate - Error creating: pods \"model-serving-api-78bbbb9bf8-\" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount \"model-serving-api\" not found"],"confidence":"medium"} 2026/07/08 19:26:58 Diagnostics complete for failed test: components controller_test.go:491: Stopping: components test failed. 2026/07/08 19:26:58 Circuit breaker: failure classified as infrastructure/rbac (code=1011) 2026/07/08 19:26:58 Circuit breaker: infrastructure/unknown failure recorded (1/3) Deletion Policy: Never. Skipping deletion tests. 2026/07/08 19:27:00 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (1851.59s)