=== RUN TestOdhOperator/components/group_2/kueue/Validate_component_removed_to_unmanaged_transition helper_test.go:150: Setting Kueue component (default-kueue) to Removed mode to start with a clean state. 2026-07-12T18:55:18Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.20.12"}} helper_test.go:150: Cleaning up any existing Kueue test resources. helper_test.go:150: Cleaning up Kueue cluster-scoped resources helper_test.go:150: Will attempt to delete 2 Kueue resources helper_test.go:150: Attempting to delete ClusterQueue /default helper_test.go:150: Successfully processed deletion of ClusterQueue /default helper_test.go:150: Attempting to delete Kueue /cluster helper_test.go:150: Successfully processed deletion of Kueue /cluster helper_test.go:150: Creating test namespace (test-kueue-managed-d99u65h86q2s7ba4vvsg) with Kueue management annotation. helper_test.go:150: Creating Kueue ConfigMap (kueue-manager-config) to verify default Kueue resource is created correctly. helper_test.go:150: Creating Kueue ConfigMap (kueue-manager-config) in namespace (redhat-ods-applications). helper_test.go:150: Updating the management state of the component in the DataScienceCluster to Unmanaged (expecting NotReady initially due to missing operator). helper_test.go:150: Ensuring LWS Operator is installed (namespace=openshift-lws-operator, name=leader-worker-set). helper_test.go:150: Ensuring LeaderWorkerSetOperator CR exists (namespace=openshift-lws-operator, name=cluster). helper_test.go:150: Ensuring OCP Kueue Operator is installed (namespace=openshift-kueue-operator, name=kueue-operator). helper_test.go:150: Verifying the component in the DataScienceCluster transitions to Ready state. helper_test.go:150: Verifying Kueue configuration (cluster) is created with all expected frameworks. testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_2/kueue/Validate_component_removed_to_unmanaged_transition (31.18s) === RUN TestOdhOperator/components/group_2/kueue === PAUSE TestOdhOperator/components/group_2/kueue === CONT TestOdhOperator/components/group_2/kueue test_context_test.go:223: Timed out after 10.000s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:114 with: Expected resource 'cluster' of kind 'Kueue' to match condition {(["BatchJob", "Deployment", "JobSet", "LeaderWorkerSet", "MPIJob", "PaddleJob", "Pod", "PyTorchJob", "RayCluster", "RayJob", "StatefulSet", "TFJob", "TrainJob", "XGBoostJob"] - .spec.config.integrations.frameworks) | length == 0 []}. Expected <string>: map[apiVersion:kueue.openshift.io/v1 kind:Kueue metadata:map[annotations:map[platform.opendatahub.io/instance.generation:1 platform.opendatahub.io/instance.name:default-kueue platform.opendatahub.io/instance.uid:34653fd1-8e8d-49c6-a522-8d2b59ad98d5 platform.opendatahub.io/type:OpenShift AI Self-Managed platform.opendatahub.io/version:3.5.0-ea.2] creationTimestamp:2026-07-12T17:58:11Z deletionGracePeriodSeconds:0 deletionTimestamp:2026-07-12T18:55:29Z finalizers:[kueue.openshift.io/finalizer] generation:2 labels:map[platform.opendatahub.io/part-of:kueue] managedFields:[map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:status:map[f:conditions:map[k:{"type":"Available"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"CertManagerAvailable"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"Degraded"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"Progressing"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]]] f:readyReplicas:map[]]] manager:kueue-operator operation:Apply subresource:status time:2026-07-12T17:59:03Z] map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[.:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:labels:map[.:map[] f:platform.opendatahub.io/part-of:map[]]] f:spec:map[.:map[] f:config:map[.:map[] f:integrations:map[.:map[] f:frameworks:map[.:map[] v:"Deployment":map[] v:"Pod":map[] v:"PyTorchJob":map[] v:"RayCluster":map[] v:"RayJob":map[] v:"StatefulSet":map[] v:"TrainJob":map[]]]] f:logLevel:map[] f:managementState:map[] f:operatorLogLevel:map[]]] manager:manager operation:Update time:2026-07-12T17:58:11Z] map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:finalizers:map[.:map[] v:"kueue.openshift.io/finalizer":map[]]]] manager:kueue-operator operation:Update time:2026-07-12T18:55:29Z]] name:cluster resourceVersion:212851980 uid:aefe5008-c06d-4317-a098-7618a0f26daf] spec:map[config:map[integrations:map[frameworks:[Deployment Pod PyTorchJob RayCluster RayJob StatefulSet TrainJob]]] logLevel:Normal managementState:Managed operatorLogLevel:Normal] status:map[conditions:[map[lastTransitionTime:2026-07-12T17:59:03Z message:All 2 replicas are ready reason:AllReplicasReady status:True type:Available] map[lastTransitionTime:2026-07-12T17:59:03Z message:Deployment is up to date reason:AsExpected status:False type:Progressing] map[lastTransitionTime:2026-07-12T17:59:03Z message: reason:AsExpected status:False type:Degraded] map[lastTransitionTime:2026-07-12T17:58:36Z message:cert-manager is installed reason:CertManagerInstalled status:True type:CertManagerAvailable]] readyReplicas:2]] to match expression <string>: (["BatchJob", "Deployment", "JobSet", "LeaderWorkerSet", "MPIJob", "PaddleJob", "Pod", "PyTorchJob", "RayCluster", "RayJob", "StatefulSet", "TFJob", "TrainJob", "XGBoostJob"] - .spec.config.integrations.frameworks) | length == 0 test_context_test.go:1106: cleanup: best-effort delete failed for /, Kind= redhat-ods-applications/kueue-manager-config: Object 'Kind' is missing in 'unstructured object has no kind' --- FAIL: TestOdhOperator/components/group_2/kueue (31.35s) === RUN TestOdhOperator/components/group_2 --- FAIL: TestOdhOperator/components/group_2 (0.00s) === RUN TestOdhOperator/components/group_4/modelsasservice === PAUSE TestOdhOperator/components/group_4/modelsasservice === CONT TestOdhOperator/components/group_4/modelsasservice 2026-07-12T19:03:10Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.20.12"}} controller_test.go:753: Creating MaaS PostgreSQL instance in namespace redhat-ods-applications controller_test.go:753: MaaS PostgreSQL instance and maas-db-config secret created in namespace redhat-ods-applications controller_test.go:753: Creating MaaS GatewayClass and Gateway: openshift-ingress/maas-default-gateway controller_test.go:753: Using hostname=maas.apps.rosa.rh-nightly-pm.nuth.p3.openshiftapps.com, certSecret=2o8clk8b6cj832lmkd2oj78crnf7qn6d-primary-cert-bundle-secret controller_test.go:753: MaaS Gateway openshift-ingress/maas-default-gateway created with HTTP+HTTPS listeners resource_applier_test.go:202: Timed out after 600.008s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:114 with: Expected resource 'default-dsc' of kind 'DataScienceCluster' to match condition {.status.conditions[] | select(.type == "KserveReady") | .status == "True" []}. Expected <string>: map[apiVersion:datasciencecluster.opendatahub.io/v2 kind:DataScienceCluster metadata:map[annotations:map[kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"datasciencecluster.opendatahub.io/v2","kind":"DataScienceCluster","metadata":{"annotations":{},"name":"default-dsc"},"spec":{"components":{"aipipelines":{"managementState":"Managed"},"codeflare":{"managementState":"Managed"},"dashboard":{"managementState":"Managed"},"feastoperator":{"managementState":"Managed"},"kserve":{"defaultDeploymentMode":"RawDeployment","managementState":"Managed","serving":{"managementState":"Removed","name":"knative-serving"}},"kueue":{"managementState":"Removed"},"llamastackoperator":{"managementState":"Removed"},"mlflowoperator":{"managementState":"Managed"},"modelregistry":{"managementState":"Managed"},"ogx":{"managementState":"Removed"},"ray":{"managementState":"Managed"},"sparkoperator":{"managementState":"Removed"},"trainer":{"managementState":"Managed"},"trainingoperator":{"managementState":"Removed"},"trustyai":{"managementState":"Managed"},"workbenches":{"managementState":"Managed"}}}} ] creationTimestamp:2026-07-12T17:42:39Z generation:56 managedFields:[map[apiVersion:datasciencecluster.opendatahub.io/v2 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:kubectl.kubernetes.io/last-applied-configuration:map[]]] f:status:map[f:components:map[f:aigateway:map[] f:aipipelines:map[f:managementState:map[] f:releases:map[k:{"name":"Kubeflow Pipelines"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:dashboard:map[f:managementState:map[]] f:feastoperator:map[f:managementState:map[] f:releases:map[k:{"name":"Feast"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:kserve:map[f:managementState:map[] f:releases:map[k:{"name":"KServe"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-router"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-workload-variant-autoscaler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"vLLM"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:kueue:map[f:managementState:map[]] f:llamastackoperator:map[] f:mlflowoperator:map[f:managementState:map[] f:releases:map[k:{"name":"MLflow"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:modelregistry:map[f:managementState:map[] f:registriesNamespace:map[] f:releases:map[k:{"name":"Kubeflow Model Registry"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"Open Data Hub Model Registry Operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:ogx:map[f:managementState:map[] f:releases:map[k:{"name":"OGX Operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"OGX"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:ray:map[f:managementState:map[] f:releases:map[k:{"name":"KubeRay"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:sparkoperator:map[f:managementState:map[] f:releases:map[k:{"name":"Spark Operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"Spark"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:trainer:map[f:managementState:map[] f:releases:map[k:{"name":"Kubeflow Trainer"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:trainingoperator:map[f:managementState:map[] f:releases:map[k:{"name":"Kubeflow Training Operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:trustyai:map[f:managementState:map[] f:releases:map[k:{"name":"EvalHub"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI Guardrails orchestrator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI LMEval driver"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI LMEval job"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI builtin detectors"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI service"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI sidecar gateway"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:workbenches:map[f:managementState:map[] f:releases:map[k:{"name":"Kubeflow Notebook Controller"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]] f:workbenchNamespace:map[]]] f:release:map[f:name:map[] f:version:map[]]]] manager:modules operation:Apply subresource:status time:2026-07-12T19:06:46Z] map[apiVersion:datasciencecluster.opendatahub.io/v2 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:kubectl.kubernetes.io/last-applied-configuration:map[]]] f:status:map[f:components:map[f:aigateway:map[] f:aipipelines:map[f:managementState:map[]] f:dashboard:map[f:managementState:map[]] f:feastoperator:map[f:managementState:map[]] f:kserve:map[f:managementState:map[] f:releases:map[k:{"name":"KServe"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-router"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-workload-variant-autoscaler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"vLLM"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:kueue:map[f:managementState:map[]] f:llamastackoperator:map[] f:mlflowoperator:map[f:managementState:map[]] f:modelregistry:map[f:managementState:map[]] f:ogx:map[f:managementState:map[]] f:ray:map[f:managementState:map[]] f:sparkoperator:map[f:managementState:map[]] f:trainer:map[f:managementState:map[]] f:trainingoperator:map[f:managementState:map[]] f:trustyai:map[f:managementState:map[]] f:workbenches:map[f:managementState:map[]]] f:conditions:map[] f:observedGeneration:map[] f:phase:map[] f:release:map[f:name:map[] f:version:map[]]]] manager:datasciencecluster operation:Apply subresource:status time:2026-07-12T19:07:09Z] map[apiVersion:datasciencecluster.opendatahub.io/v2 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[.:map[] f:kubectl.kubernetes.io/last-applied-configuration:map[]]] f:spec:map[.:map[] f:components:map[.:map[] f:aipipelines:map[] f:dashboard:map[] f:feastoperator:map[] f:kserve:map[.:map[] f:nim:map[.:map[] f:airGapped:map[] f:managementState:map[]] f:rawDeploymentServiceConfig:map[]] f:kueue:map[.:map[] f:defaultClusterQueueName:map[] f:defaultLocalQueueName:map[]] f:llamastackoperator:map[.:map[] f:managementState:map[]] f:mlflowoperator:map[] f:modelregistry:map[.:map[] f:registriesNamespace:map[]] f:ogx:map[] f:ray:map[] f:sparkoperator:map[] f:trainer:map[] f:trainingoperator:map[] f:trustyai:map[.:map[] f:mcpGuardrailsMode:map[]] f:workbenches:map[.:map[] f:workbenchNamespace:map[]]]]] manager:kubectl-client-side-apply operation:Update time:2026-07-12T17:42:39Z] map[apiVersion:datasciencecluster.opendatahub.io/v2 fieldsType:FieldsV1 fieldsV1:map[f:spec:map[f:components:map[f:kserve:map[f:modelsAsService:map[.:map[] f:managementState:map[]]]]]] manager:kubectl-patch operation:Update time:2026-07-12T17:58:08Z] map[apiVersion:datasciencecluster.opendatahub.io/v2 fieldsType:FieldsV1 fieldsV1:map[f:spec:map[f:components:map[f:aigateway:map[.:map[] f:managementState:map[]] f:aipipelines:map[f:managementState:map[]] f:dashboard:map[f:managementState:map[]] f:feastoperator:map[f:managementState:map[]] f:kserve:map[f:managementState:map[] f:wva:map[.:map[] f:managementState:map[]]] f:kueue:map[f:managementState:map[]] f:mlflowoperator:map[f:managementState:map[]] f:modelregistry:map[f:managementState:map[]] f:ogx:map[f:managementState:map[]] f:ray:map[f:managementState:map[]] f:sparkoperator:map[f:managementState:map[]] f:trainer:map[f:managementState:map[]] f:trainingoperator:map[f:managementState:map[]] f:trustyai:map[f:managementState:map[]] f:workbenches:map[f:managementState:map[]]]]] manager:e2e-tests operation:Update time:2026-07-12T19:03:20Z]] name:default-dsc resourceVersion:212867380 uid:3f2b4ae3-e88d-428c-8c0e-25f8e753404f] spec:map[components:map[aigateway:map[managementState:Removed] aipipelines:map[managementState:Removed] dashboard:map[managementState:Removed] feastoperator:map[managementState:Removed] kserve:map[managementState:Managed modelsAsService:map[managementState:Managed] nim:map[airGapped:false managementState:Managed] rawDeploymentServiceConfig:Headless wva:map[managementState:Removed]] kueue:map[defaultClusterQueueName:default defaultLocalQueueName:default managementState:Unmanaged] llamastackoperator:map[managementState:Removed] mlflowoperator:map[managementState:Removed] modelregistry:map[managementState:Removed registriesNamespace:rhoai-model-registries] ogx:map[managementState:Removed] ray:map[managementState:Removed] sparkoperator:map[managementState:Removed] trainer:map[managementState:Removed] trainingoperator:map[managementState:Removed] trustyai:map[managementState:Removed mcpGuardrailsMode:false] workbenches:map[managementState:Removed workbenchNamespace:rhods-notebooks]]] status:map[components:map[aigateway:map[] aipipelines:map[managementState:Removed releases:[map[name:Kubeflow Pipelines repoUrl:https://github.com/kubeflow/pipelines version:2.16.0]]] dashboard:map[managementState:Removed] feastoperator:map[managementState:Removed releases:[map[name:Feast repoUrl:https://github.com/feast-dev/feast version:0.64.0]]] kserve:map[managementState:Managed releases:[map[name:KServe repoUrl:https://github.com/kserve/kserve/ version:v0.19.0] map[name:vLLM repoUrl:https://github.com/vllm-project/vllm version:v0.21.0] map[name:llm-d-router repoUrl:https://github.com/llm-d/llm-d-router version:v0.9.0] map[name:llm-d-workload-variant-autoscaler repoUrl:https://github.com/llm-d/llm-d-workload-variant-autoscaler version:v0.8.0]]] kueue:map[managementState:Unmanaged] llamastackoperator:map[] mlflowoperator:map[managementState:Removed releases:[map[name:MLflow repoUrl:https://github.com/mlflow/mlflow version:v3.13.0]]] modelregistry:map[managementState:Removed registriesNamespace:rhoai-model-registries releases:[map[name:Kubeflow Model Registry repoUrl:https://github.com/kubeflow/model-registry version:latest] map[name:Open Data Hub Model Registry Operator repoUrl:https://github.com/opendatahub-io/model-registry-operator version:latest]]] ogx:map[managementState:Removed releases:[map[name:OGX repoUrl:https://github.com/opendatahub-io/ogx-distribution version:v1.1.2] map[name:OGX Operator repoUrl:https://github.com/opendatahub-io/ogx-k8s-operator version:v0.11.0]]] ray:map[managementState:Removed releases:[map[name:KubeRay repoUrl:https://github.com/ray-project/kuberay version:1.4.2]]] sparkoperator:map[managementState:Removed releases:[map[name:Spark Operator repoUrl:https://github.com/opendatahub-io/spark-operator version:v2.4.0] map[name:Spark repoUrl:https://github.com/apache/spark version:4.0.1]]] trainer:map[managementState:Removed releases:[map[name:Kubeflow Trainer repoUrl:https://github.com/kubeflow/trainer version:2.1.0]]] trainingoperator:map[managementState:Removed releases:[map[name:Kubeflow Training Operator repoUrl:https://github.com/kubeflow/trainer version:1.9.0]]] trustyai:map[managementState:Removed releases:[map[name:TrustyAI operator repoUrl:https://github.com/trustyai-explainability/trustyai-service-operator version:v1.37.0] map[name:TrustyAI service repoUrl:https://github.com/trustyai-explainability/trustyai-explainability version:v0.28.0] map[name:TrustyAI LMEval driver repoUrl:https://github.com/trustyai-explainability/trustyai-service-operator version:v1.37.0] map[name:TrustyAI LMEval job repoUrl:https://github.com/EleutherAI/lm-evaluation-harness version:v0.4.8] map[name:TrustyAI Guardrails orchestrator repoUrl:https://github.com/foundation-model-stack/fms-guardrails-orchestrator version:0.9.4] map[name:TrustyAI builtin detectors repoUrl:https://github.com/trustyai-explainability/guardrails-regex-detector version:v0.2.0] map[name:TrustyAI sidecar gateway repoUrl:https://github.com/trustyai-explainability/vllm-orchestrator-gateway version:v0.2.1] map[name:EvalHub repoUrl:https://github.com/eval-hub/eval-hub version:v0.3.0]]] workbenches:map[managementState:Removed releases:[map[name:Kubeflow Notebook Controller repoUrl:https://github.com/kubeflow/kubeflow version:1.10.0]] workbenchNamespace:rhods-notebooks]] conditions:[map[lastTransitionTime:2026-07-12T19:04:34Z message:Some components are not ready: kserve reason:NotReady status:False type:Ready] map[lastTransitionTime:2026-07-12T19:03:23Z observedGeneration:56 status:True type:ProvisioningSucceeded] map[lastTransitionTime:2026-07-12T19:04:34Z message:Some components are not ready: kserve reason:NotReady status:False type:ComponentsReady] map[lastTransitionTime:2026-07-12T19:07:09Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:AIPipelinesReady] map[lastTransitionTime:2026-07-12T19:07:09Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:DashboardReady] map[lastTransitionTime:2026-07-12T19:07:09Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:FeastOperatorReady] map[lastTransitionTime:2026-07-12T19:03:46Z status:True type:KserveLLMInferenceServiceDependencies] map[lastTransitionTime:2026-07-12T19:03:46Z status:True type:KserveLLMInferenceServiceWideEPDependencies] map[lastTransitionTime:2026-07-12T19:07:09Z message:1/2 deployments ready reason:DeploymentsNotReady status:False type:KserveReady] map[lastTransitionTime:2026-07-12T19:07:09Z status:True type:KueueReady] map[lastTransitionTime:2026-07-12T19:03:28Z status:True type:LLM-D-WVADependencies] map[lastTransitionTime:2026-07-12T19:07:09Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:MLflowOperatorReady] map[lastTransitionTime:2026-07-12T19:03:22Z message:maas-default-gateway is missing required annotations: security.opendatahub.io/authorino-tls-bootstrap="true" reason:MaaSPrerequisitesNotMet severity:Info status:False type:MaaSPrerequisitesAvailable] map[lastTransitionTime:2026-07-12T19:07:09Z status:True type:ModelControllerReady] map[lastTransitionTime:2026-07-12T19:07:09Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:ModelRegistryReady] map[lastTransitionTime:2026-07-12T19:07:09Z message:MaaS platform manifests applied and maas-api deployment is available reason:Reconciled status:True type:ModelsAsServiceReady] map[lastTransitionTime:2026-07-12T19:07:09Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:OGXReady] map[lastTransitionTime:2026-07-12T19:07:09Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:RayReady] map[lastTransitionTime:2026-07-12T19:07:09Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:SparkOperatorReady] map[lastTransitionTime:2026-07-12T19:07:09Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:TrainerReady] map[lastTransitionTime:2026-07-12T19:07:09Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:TrainingOperatorReady] map[lastTransitionTime:2026-07-12T19:07:09Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:TrustyAIReady] map[lastTransitionTime:2026-07-12T19:07:09Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:WorkbenchesReady]] observedGeneration:55 phase:Not Ready release:map[name:OpenShift AI Self-Managed version:3.5.0-ea.2]]] to match expression <string>: .status.conditions[] | select(.type == "KserveReady") | .status == "True" --- FAIL: TestOdhOperator/components/group_4/modelsasservice (610.51s) === RUN TestOdhOperator/components/group_4 --- FAIL: TestOdhOperator/components/group_4 (0.00s) === RUN TestOdhOperator/components 2026/07/12 18:59:00 TEST FAILURE DETECTED: group 2 2026/07/12 18:59:00 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 2) === 2026/07/12 18:59:01 === CLUSTER STATE === 2026/07/12 18:59:01 Node: ip-10-0-1-120.us-east-2.compute.internal 2026/07/12 18:59:01 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 18:59:01 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 18:59:01 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 18:59:01 Condition Ready: True - kubelet is posting ready status 2026/07/12 18:59:01 Resources: 7500m CPU, 31236228Ki memory allocatable (of 8000m CPU, 32387204Ki memory capacity) 2026/07/12 18:59:01 Node: ip-10-0-1-132.us-east-2.compute.internal 2026/07/12 18:59:01 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 18:59:01 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 18:59:01 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 18:59:01 Condition Ready: True - kubelet is posting ready status 2026/07/12 18:59:01 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/12 18:59:01 Node: ip-10-0-1-39.us-east-2.compute.internal 2026/07/12 18:59:01 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 18:59:01 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 18:59:01 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 18:59:01 Condition Ready: True - kubelet is posting ready status 2026/07/12 18:59:01 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/12 18:59:01 === DEPLOYMENTS === 2026/07/12 18:59:01 Namespace: redhat-ods-applications 2026/07/12 18:59:01 Deployment maas-api: 1/1 (READY) 2026/07/12 18:59:01 Deployment maas-controller: 1/1 (READY) 2026/07/12 18:59:01 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/12 18:59:01 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/12 18:59:01 Namespace: kube-system 2026/07/12 18:59:01 === PODS === 2026/07/12 18:59:01 Namespace redhat-ods-applications: no problematic pods found 2026/07/12 18:59:01 Namespace: kube-system 2026/07/12 18:59:01 Pod ecr-credential-refresh-initial-k6gr5: Succeeded 2026/07/12 18:59:01 Container refresh: ready=false, terminated: Completed (exit 0) 2026/07/12 18:59:01 === LOGS (CURRENT) for container refresh === 2026/07/12 18:59:01 2026-04-23 02:42:29 - Discovering ECR registries from cluster configuration... 2026/07/12 18:59:01 2026-04-23 02:42:31 - No IDMS mirrors found 2026/07/12 18:59:01 2026-04-23 02:42:31 - Skipping - no ECR IDMS mirrors found (not a zero-egress cluster) 2026/07/12 18:59:01 === END LOGS === 2026/07/12 18:59:01 === OPERATOR STATUS === 2026/07/12 18:59:01 Operator deployment: 3/3 ready 2026/07/12 18:59:01 === DSCI STATUS === 2026/07/12 18:59:01 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/12 18:59:01 === DSC STATUS === 2026/07/12 18:59:01 Failed to collect DSC data: default-dsc unhealthy conditions: ModelsAsServiceReady=False 2026/07/12 18:59:01 === RECENT EVENTS === 2026/07/12 18:59:01 18:58:53 INFO Pod/odh-model-controller-867855946f-q9f57: Killing - Stopping container manager 2026/07/12 18:59:01 18:58:53 INFO Pod/model-serving-api-6688cd56bd-cvw8b: Killing - Stopping container server 2026/07/12 18:59:01 18:58:53 INFO Pod/model-registry-operator-controller-manager-66d8c86b58-nhlmg: Killing - Stopping container manager 2026/07/12 18:59:01 18:58:53 WARN Pod/odh-model-controller-867855946f-q9f57: Unhealthy - Readiness probe failed: Get "http://10.130.11.3:8081/readyz": dial tcp 10.130.11.3:8081: connect: connection refused 2026/07/12 18:59:01 18:58:52 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-kmm6j: Killing - Stopping container manager 2026/07/12 18:59:01 18:58:52 INFO Pod/kserve-controller-manager-6c67dcb69c-l48dv: Killing - Stopping container manager 2026/07/12 18:59:01 18:58:52 INFO Pod/llmisvc-controller-manager-5d65df97bb-w54l6: Killing - Stopping container manager 2026/07/12 18:59:01 18:58:51 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/12 18:59:01 18:58:38 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-867855946f-q9f57_981bd041-8000-4b55-abfc-b7343545bbcd became leader 2026/07/12 18:59:01 18:58:36 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-kmm6j: Pulled - Container image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" already present on machine 2026/07/12 18:59:01 18:58:36 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-kmm6j: Started - Started container manager 2026/07/12 18:59:01 18:58:36 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-kmm6j: Created - Created container: manager 2026/07/12 18:59:01 18:58:36 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-kmm6j: AddedInterface - Add eth0 [10.130.11.4/23] from ovn-kubernetes 2026/07/12 18:59:01 18:58:36 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-5bd688895b-kmm6j_e5eeec0c-b50a-477c-a26b-fe050e87b53c became leader 2026/07/12 18:59:01 18:58:35 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-kmm6j: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-5bd688895b-kmm6j to ip-10-0-1-39.us-east-2.compute.internal 2026/07/12 18:59:01 18:58:35 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-5bd688895b-kmm6j 2026/07/12 18:59:01 18:58:35 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-bx4cm: Killing - Stopping container manager 2026/07/12 18:59:01 18:58:35 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-cw7f2: Killing - Stopping container manager 2026/07/12 18:59:01 18:58:35 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-5bd688895b from 0 to 1 2026/07/12 18:59:01 18:58:21 INFO Pod/odh-model-controller-867855946f-q9f57: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 511ms (511ms including waiting). Image size: 238959091 bytes. 2026/07/12 18:59:01 18:58:21 INFO Pod/odh-model-controller-867855946f-q9f57: Started - Started container manager 2026/07/12 18:59:01 18:58:21 INFO Pod/odh-model-controller-867855946f-q9f57: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/12 18:59:01 18:58:21 INFO Pod/odh-model-controller-867855946f-q9f57: AddedInterface - Add eth0 [10.130.11.3/23] from ovn-kubernetes 2026/07/12 18:59:01 18:58:21 INFO Pod/odh-model-controller-867855946f-q9f57: Created - Created container: manager 2026/07/12 18:59:01 18:58:20 INFO Pod/odh-model-controller-867855946f-q9f57: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-867855946f-q9f57 to ip-10-0-1-39.us-east-2.compute.internal 2026/07/12 18:59:01 18:58:20 INFO Pod/odh-model-controller-867855946f-plgm8: Killing - Stopping container manager 2026/07/12 18:59:01 18:58:20 INFO Pod/odh-model-controller-867855946f-kh92j: Killing - Stopping container manager 2026/07/12 18:59:01 18:58:20 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-867855946f from 0 to 1 2026/07/12 18:59:01 18:58:20 INFO ReplicaSet/odh-model-controller-867855946f: SuccessfulCreate - Created pod: odh-model-controller-867855946f-q9f57 2026/07/12 18:59:01 18:58:06 INFO Pod/model-serving-api-6688cd56bd-cvw8b: Created - Created container: server 2026/07/12 18:59:01 18:58:06 INFO Pod/model-serving-api-6688cd56bd-cvw8b: Started - Started container server 2026/07/12 18:59:01 18:58:06 INFO Pod/model-serving-api-6688cd56bd-cvw8b: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/12 18:59:01 18:58:05 INFO Pod/model-serving-api-6688cd56bd-cvw8b: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-cvw8b to ip-10-0-1-39.us-east-2.compute.internal 2026/07/12 18:59:01 18:58:05 INFO Pod/model-serving-api-6688cd56bd-p57xl: Killing - Stopping container server 2026/07/12 18:59:01 18:58:05 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-cvw8b 2026/07/12 18:59:01 18:58:05 INFO Pod/model-serving-api-6688cd56bd-k9rxt: Killing - Stopping container server 2026/07/12 18:59:01 18:58:05 INFO Pod/model-serving-api-6688cd56bd-cvw8b: AddedInterface - Add eth0 [10.130.11.2/23] from ovn-kubernetes 2026/07/12 18:59:01 18:58:05 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/12 18:59:01 18:57:55 WARN Pod/odh-model-controller-867855946f-kh92j: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/12 18:59:01 18:57:05 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-bx4cm: Started - Started container manager 2026/07/12 18:59:01 18:57:05 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-bx4cm: AddedInterface - Add eth0 [10.130.11.1/23] from ovn-kubernetes 2026/07/12 18:59:01 18:57:05 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-bx4cm: Created - Created container: manager 2026/07/12 18:59:01 18:57:05 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-bx4cm: Pulled - Container image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" already present on machine 2026/07/12 18:59:01 18:57:04 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-bx4cm: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-5bd688895b-bx4cm to ip-10-0-1-39.us-east-2.compute.internal 2026/07/12 18:59:01 18:57:04 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-5bd688895b-bx4cm 2026/07/12 18:59:01 18:57:04 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-5bd688895b from 1 to 2 2026/07/12 18:59:01 18:56:45 INFO Pod/odh-model-controller-867855946f-plgm8: AddedInterface - Add eth0 [10.128.0.102/23] from ovn-kubernetes 2026/07/12 18:59:01 18:56:45 INFO Pod/odh-model-controller-867855946f-plgm8: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/12 18:59:01 18:56:45 INFO Pod/odh-model-controller-867855946f-plgm8: Created - Created container: manager 2026/07/12 18:59:01 18:56:45 INFO Pod/odh-model-controller-867855946f-plgm8: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 422ms (422ms including waiting). Image size: 238959091 bytes. 2026/07/12 18:59:01 18:56:45 INFO Pod/odh-model-controller-867855946f-plgm8: Started - Started container manager 2026/07/12 18:59:01 18:56:44 INFO Pod/odh-model-controller-867855946f-plgm8: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-867855946f-plgm8 to ip-10-0-1-120.us-east-2.compute.internal 2026/07/12 18:59:01 18:56:44 INFO ReplicaSet/odh-model-controller-867855946f: SuccessfulCreate - Created pod: odh-model-controller-867855946f-plgm8 2026/07/12 18:59:01 18:56:44 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-867855946f from 1 to 2 2026/07/12 18:59:01 18:56:25 INFO Pod/model-serving-api-6688cd56bd-p57xl: Created - Created container: server 2026/07/12 18:59:01 18:56:25 INFO Pod/model-serving-api-6688cd56bd-p57xl: Started - Started container server 2026/07/12 18:59:01 18:56:25 INFO Pod/model-serving-api-6688cd56bd-p57xl: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/12 18:59:01 18:56:25 INFO Pod/model-serving-api-6688cd56bd-p57xl: AddedInterface - Add eth0 [10.128.0.101/23] from ovn-kubernetes 2026/07/12 18:59:01 18:56:24 INFO Pod/model-serving-api-6688cd56bd-p57xl: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-p57xl to ip-10-0-1-120.us-east-2.compute.internal 2026/07/12 18:59:01 18:56:24 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 1 to 2 2026/07/12 18:59:01 18:56:24 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-p57xl 2026/07/12 18:59:01 18:56:13 INFO Pod/maas-api-84449bdd5d-5fmk9: Created - Created container: maas-api 2026/07/12 18:59:01 18:56:13 INFO Pod/maas-api-84449bdd5d-5fmk9: Started - Started container maas-api 2026/07/12 18:59:01 18:56:13 INFO Pod/maas-api-84449bdd5d-5fmk9: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" in 498ms (498ms including waiting). Image size: 175018396 bytes. 2026/07/12 18:59:01 18:56:12 INFO Pod/maas-api-84449bdd5d-5fmk9: Scheduled - Successfully assigned redhat-ods-applications/maas-api-84449bdd5d-5fmk9 to ip-10-0-1-120.us-east-2.compute.internal 2026/07/12 18:59:01 18:56:12 INFO Pod/maas-api-84449bdd5d-5fmk9: AddedInterface - Add eth0 [10.128.0.98/23] from ovn-kubernetes 2026/07/12 18:59:01 18:56:12 WARN ReplicaSet/maas-api-84449bdd5d: FailedCreate - Error creating: pods "maas-api-84449bdd5d-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 2026/07/12 18:59:01 18:56:12 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/12 18:59:01 18:56:12 INFO Deployment/maas-api: ScalingReplicaSet - Scaled up replica set maas-api-84449bdd5d from 0 to 1 2026/07/12 18:59:01 18:56:12 INFO ReplicaSet/maas-api-84449bdd5d: SuccessfulCreate - Created pod: maas-api-84449bdd5d-5fmk9 2026/07/12 18:59:01 18:56:12 INFO Pod/maas-api-84449bdd5d-5fmk9: Pulling - Pulling image "quay.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" 2026/07/12 18:59:01 18:55:58 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-w54l6_d05a2d14-c643-423b-bf20-99d9197f2720 became leader 2026/07/12 18:59:01 18:55:55 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-l48dv_eefd14b9-ada8-4ade-9829-1e47dc959d55 became leader 2026/07/12 18:59:01 18:55:45 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-5bd688895b-cw7f2_e82bde94-4db0-4f6e-b0ec-a3f82085f0bc became leader 2026/07/12 18:59:01 18:55:44 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-cw7f2: Created - Created container: manager 2026/07/12 18:59:01 18:55:44 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-cw7f2: AddedInterface - Add eth0 [10.128.0.94/23] from ovn-kubernetes 2026/07/12 18:59:01 18:55:44 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-cw7f2: Started - Started container manager 2026/07/12 18:59:01 18:55:44 WARN ServiceMonitor/workload-variant-autoscaler-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/workload-variant-autoscaler-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: bearerTokenSecret: [REDACTED] to get secret "workload-variant-autoscaler-controller-manager-token": secrets "workload-variant-autoscaler-controller-manager-token" not found 2026/07/12 18:59:01 18:55:44 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-cw7f2: Pulled - Container image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" already present on machine 2026/07/12 18:59:01 18:55:43 WARN Pod/workload-variant-autoscaler-controller-manager-5bd688895b-cw7f2: FailedMount - MountVolume.SetUp failed for volume "epp-metrics-token" : secret "workload-variant-autoscaler-epp-metrics-token" not found 2026/07/12 18:59:01 18:55:42 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-cw7f2: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-5bd688895b-cw7f2 to ip-10-0-1-120.us-east-2.compute.internal 2026/07/12 18:59:01 18:55:42 WARN ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: FailedCreate - Error creating: pods "workload-variant-autoscaler-controller-manager-5bd688895b-" is forbidden: error looking up service account redhat-ods-applications/workload-variant-autoscaler-controller-manager: serviceaccount "workload-variant-autoscaler-controller-manager" not found 2026/07/12 18:59:01 18:55:42 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-5bd688895b from 0 to 1 2026/07/12 18:59:01 18:55:42 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-5bd688895b-cw7f2 2026/07/12 18:59:01 18:55:41 INFO Pod/llmisvc-controller-manager-5d65df97bb-w54l6: Created - Created container: manager 2026/07/12 18:59:01 18:55:41 INFO Pod/llmisvc-controller-manager-5d65df97bb-w54l6: Started - Started container manager 2026/07/12 18:59:01 18:55:40 INFO Pod/kserve-controller-manager-6c67dcb69c-l48dv: Started - Started container manager 2026/07/12 18:59:01 18:55:40 INFO Pod/llmisvc-controller-manager-5d65df97bb-w54l6: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/12 18:59:01 18:55:40 INFO Lease/85f368d1.opendatahub.io: LeaderElection - model-registry-operator-controller-manager-66d8c86b58-nhlmg_5a270a08-7fc9-4b93-8689-77b3f4ebaaec became leader 2026/07/12 18:59:01 18:55:40 INFO Pod/kserve-controller-manager-6c67dcb69c-l48dv: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/12 18:59:01 18:55:40 INFO Pod/kserve-controller-manager-6c67dcb69c-l48dv: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 495ms (495ms including waiting). Image size: 238435613 bytes. 2026/07/12 18:59:01 18:55:40 INFO Pod/kserve-controller-manager-6c67dcb69c-l48dv: Created - Created container: manager 2026/07/12 18:59:01 18:55:40 INFO Lease/maas-controller.models-as-a-service.opendatahub.io: LeaderElection - maas-controller-85bdcf9988-68gf8_a9615d34-0bc7-4c33-a5d8-12a826aaa901 became leader 2026/07/12 18:59:01 18:55:40 INFO Pod/llmisvc-controller-manager-5d65df97bb-w54l6: AddedInterface - Add eth0 [10.128.0.93/23] from ovn-kubernetes 2026/07/12 18:59:01 18:55:40 INFO Pod/llmisvc-controller-manager-5d65df97bb-w54l6: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 566ms (566ms including waiting). Image size: 236366599 bytes. 2026/07/12 18:59:01 18:55:39 INFO Pod/llmisvc-controller-manager-5d65df97bb-w54l6: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-w54l6 to ip-10-0-1-120.us-east-2.compute.internal 2026/07/12 18:59:01 18:55:39 INFO Pod/kserve-controller-manager-6c67dcb69c-l48dv: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-l48dv to ip-10-0-1-39.us-east-2.compute.internal 2026/07/12 18:59:01 18:55:39 WARN Pod/llmisvc-controller-manager-5d65df97bb-w54l6: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 2026/07/12 18:59:01 18:55:39 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/12 18:59:01 18:55:39 INFO Pod/kserve-controller-manager-6c67dcb69c-l48dv: AddedInterface - Add eth0 [10.130.10.254/23] from ovn-kubernetes 2026/07/12 18:59:01 18:55:39 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-w54l6 2026/07/12 18:59:01 18:55:39 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-867855946f-kh92j_1456965d-78b5-4413-b3b9-336e5c6d034b became leader 2026/07/12 18:59:01 18:55:39 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/12 18:59:01 18:55:39 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-l48dv 2026/07/12 18:59:01 18:55:24 INFO Pod/model-serving-api-6688cd56bd-k9rxt: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-k9rxt to ip-10-0-1-39.us-east-2.compute.internal 2026/07/12 18:59:01 18:55:24 INFO Pod/model-serving-api-6688cd56bd-k9rxt: Started - Started container server 2026/07/12 18:59:01 18:55:24 INFO Pod/odh-model-controller-867855946f-kh92j: Created - Created container: manager 2026/07/12 18:59:01 18:55:24 INFO Pod/model-serving-api-6688cd56bd-k9rxt: Created - Created container: server 2026/07/12 18:59:01 18:55:24 INFO Pod/model-serving-api-6688cd56bd-k9rxt: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/12 18:59:01 18:55:24 INFO Pod/model-serving-api-6688cd56bd-k9rxt: AddedInterface - Add eth0 [10.130.10.253/23] from ovn-kubernetes 2026/07/12 18:59:01 18:55:24 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/12 18:59:01 18:55:24 INFO Pod/odh-model-controller-867855946f-kh92j: Started - Started container manager 2026/07/12 18:59:01 18:55:24 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-k9rxt 2026/07/12 18:59:01 18:55:24 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/12 18:59:01 18:55:23 INFO Pod/model-registry-operator-controller-manager-66d8c86b58-nhlmg: Started - Started container manager 2026/07/12 18:59:01 18:55:23 INFO Pod/model-registry-operator-controller-manager-66d8c86b58-nhlmg: AddedInterface - Add eth0 [10.130.10.249/23] from ovn-kubernetes 2026/07/12 18:59:01 18:55:23 INFO Pod/odh-model-controller-867855946f-kh92j: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 478ms (478ms including waiting). Image size: 238959091 bytes. 2026/07/12 18:59:01 18:55:23 INFO Pod/odh-model-controller-867855946f-kh92j: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/12 18:59:01 18:55:23 INFO Pod/odh-model-controller-867855946f-kh92j: AddedInterface - Add eth0 [10.130.10.252/23] from ovn-kubernetes 2026/07/12 18:59:01 18:55:23 INFO Pod/model-registry-operator-controller-manager-66d8c86b58-nhlmg: Created - Created container: manager 2026/07/12 18:59:01 18:55:23 INFO Pod/model-registry-operator-controller-manager-66d8c86b58-nhlmg: Pulled - Container image "quay.io/rhoai/odh-model-registry-operator-rhel9@sha256:ebcec5c43468283ed3cbab50530f939166f8f3e8c9d306e91a8107d05d7fa9d4" already present on machine 2026/07/12 18:59:01 18:55:23 INFO Pod/maas-controller-85bdcf9988-68gf8: Created - Created container: manager 2026/07/12 18:59:01 18:55:23 INFO Pod/maas-controller-85bdcf9988-68gf8: Started - Started container manager 2026/07/12 18:59:01 18:55:22 INFO Pod/odh-model-controller-867855946f-kh92j: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-867855946f-kh92j to ip-10-0-1-39.us-east-2.compute.internal 2026/07/12 18:59:01 18:55:22 INFO Pod/maas-controller-85bdcf9988-68gf8: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/12 18:59:01 18:55:22 INFO Pod/maas-controller-85bdcf9988-68gf8: AddedInterface - Add eth0 [10.130.10.250/23] from ovn-kubernetes 2026/07/12 18:59:01 18:55:22 WARN Pod/model-registry-operator-controller-manager-66d8c86b58-nhlmg: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 2026/07/12 18:59:01 18:55:22 INFO ReplicaSet/odh-model-controller-867855946f: SuccessfulCreate - Created pod: odh-model-controller-867855946f-kh92j 2026/07/12 18:59:01 18:55:22 INFO Pod/maas-controller-85bdcf9988-68gf8: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 558ms (558ms including waiting). Image size: 212866077 bytes. 2026/07/12 18:59:01 18:55:22 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-867855946f from 0 to 1 2026/07/12 18:59:01 18:55:21 INFO Pod/maas-controller-85bdcf9988-68gf8: Scheduled - Successfully assigned redhat-ods-applications/maas-controller-85bdcf9988-68gf8 to ip-10-0-1-39.us-east-2.compute.internal 2026/07/12 18:59:01 18:55:21 INFO Pod/model-registry-operator-controller-manager-66d8c86b58-nhlmg: Scheduled - Successfully assigned redhat-ods-applications/model-registry-operator-controller-manager-66d8c86b58-nhlmg to ip-10-0-1-39.us-east-2.compute.internal 2026/07/12 18:59:01 18:55:21 INFO ReplicaSet/maas-controller-85bdcf9988: SuccessfulCreate - Created pod: maas-controller-85bdcf9988-68gf8 2026/07/12 18:59:01 18:55:21 INFO Deployment/model-registry-operator-controller-manager: ScalingReplicaSet - Scaled up replica set model-registry-operator-controller-manager-66d8c86b58 from 0 to 1 2026/07/12 18:59:01 18:55:21 INFO ReplicaSet/model-registry-operator-controller-manager-66d8c86b58: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-66d8c86b58-nhlmg 2026/07/12 18:59:01 18:55:21 INFO Deployment/maas-controller: ScalingReplicaSet - Scaled up replica set maas-controller-85bdcf9988 from 0 to 1 2026/07/12 18:59:01 18:55:10 INFO Pod/dashboard-redirect-85dbcf645f-mmmnl: Killing - Stopping container nginx 2026/07/12 18:59:01 18:55:10 INFO Pod/dashboard-redirect-85dbcf645f-7s9bq: Killing - Stopping container nginx 2026/07/12 18:59:01 18:54:28 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: Created - Created container: mlflow-ui 2026/07/12 18:59:01 18:54:28 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" in 449ms (449ms including waiting). Image size: 177808591 bytes. 2026/07/12 18:59:01 18:54:28 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" in 418ms (418ms including waiting). Image size: 177808591 bytes. 2026/07/12 18:59:01 18:54:28 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: Created - Created container: mlflow-ui 2026/07/12 18:59:01 18:54:28 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: Started - Started container mlflow-ui 2026/07/12 18:59:01 18:54:28 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" 2026/07/12 18:59:01 18:54:28 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" in 498ms (498ms including waiting). Image size: 185092464 bytes. 2026/07/12 18:59:01 18:54:28 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" in 470ms (470ms including waiting). Image size: 185092464 bytes. 2026/07/12 18:59:01 18:54:28 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" 2026/07/12 18:59:01 18:54:28 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: Started - Started container mlflow-ui 2026/07/12 18:59:01 18:54:27 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" 2026/07/12 18:59:01 18:54:27 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" 2026/07/12 18:59:01 18:54:27 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: Created - Created container: maas-ui 2026/07/12 18:59:01 18:54:27 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" in 473ms (473ms including waiting). Image size: 187802205 bytes. 2026/07/12 18:59:01 18:54:27 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" 2026/07/12 18:59:01 18:54:27 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: Created - Created container: gen-ai-ui 2026/07/12 18:59:01 18:54:27 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: Started - Started container gen-ai-ui 2026/07/12 18:59:01 18:54:27 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" 2026/07/12 18:59:01 18:54:27 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" in 504ms (504ms including waiting). Image size: 187802205 bytes. 2026/07/12 18:59:01 18:54:27 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: Created - Created container: maas-ui 2026/07/12 18:59:01 18:54:27 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: Started - Started container maas-ui 2026/07/12 18:59:01 18:54:27 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: Started - Started container maas-ui 2026/07/12 18:59:01 18:54:27 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: Started - Started container gen-ai-ui 2026/07/12 18:59:01 18:54:27 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: Created - Created container: gen-ai-ui 2026/07/12 18:59:01 18:54:26 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: Created - Created container: model-registry-ui 2026/07/12 18:59:01 18:54:26 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: Created - Created container: model-registry-ui 2026/07/12 18:59:01 18:54:26 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: Started - Started container model-registry-ui 2026/07/12 18:59:01 18:54:26 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" 2026/07/12 18:59:01 18:54:26 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" in 548ms (548ms including waiting). Image size: 241242949 bytes. 2026/07/12 18:59:01 18:54:26 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" in 574ms (574ms including waiting). Image size: 182438697 bytes. 2026/07/12 18:59:01 18:54:26 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" in 417ms (417ms including waiting). Image size: 182438697 bytes. 2026/07/12 18:59:01 18:54:26 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" in 560ms (560ms including waiting). Image size: 241242949 bytes. 2026/07/12 18:59:01 18:54:26 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" 2026/07/12 18:59:01 18:54:26 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: Started - Started container model-registry-ui 2026/07/12 18:59:01 18:54:25 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: Started - Started container kube-rbac-proxy 2026/07/12 18:59:01 18:54:25 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: Created - Created container: rhods-dashboard 2026/07/12 18:59:01 18:54:25 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: Started - Started container rhods-dashboard 2026/07/12 18:59:01 18:54:25 INFO Pod/spark-operator-webhook-9cb9bdf44-d85tl: Killing - Stopping container webhook 2026/07/12 18:59:01 18:54:25 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: Pulled - Container image "quay.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:1e614e46799fd020ac18d95c06a463421154ce86e59ccb3fa6007e66fed9910e" already present on machine 2026/07/12 18:59:01 18:54:25 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: Created - Created container: kube-rbac-proxy 2026/07/12 18:59:01 18:54:25 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: Started - Started container kube-rbac-proxy 2026/07/12 18:59:01 18:54:25 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" 2026/07/12 18:59:01 18:54:25 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: Pulled - Successfully pulled image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" in 528ms (528ms including waiting). Image size: 799822033 bytes. 2026/07/12 18:59:01 18:54:25 INFO Pod/spark-operator-controller-767899d44f-x4sf4: Killing - Stopping container controller 2026/07/12 18:59:01 18:54:25 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" 2026/07/12 18:59:01 18:54:25 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: Created - Created container: kube-rbac-proxy 2026/07/12 18:59:01 18:54:25 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: AddedInterface - Add eth0 [10.128.0.92/23] from ovn-kubernetes 2026/07/12 18:59:01 18:54:25 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: Pulled - Container image "quay.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:1e614e46799fd020ac18d95c06a463421154ce86e59ccb3fa6007e66fed9910e" already present on machine 2026/07/12 18:59:01 18:54:25 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: Started - Started container rhods-dashboard 2026/07/12 18:59:01 18:54:25 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: Created - Created container: rhods-dashboard 2026/07/12 18:59:01 18:54:25 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: Pulled - Successfully pulled image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" in 421ms (421ms including waiting). Image size: 799822033 bytes. 2026/07/12 18:59:01 18:54:25 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: Pulling - Pulling image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" 2026/07/12 18:59:01 18:54:24 INFO Pod/rhods-dashboard-5998574cd9-vcwpc: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-5998574cd9-vcwpc to ip-10-0-1-120.us-east-2.compute.internal 2026/07/12 18:59:01 18:54:24 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-5998574cd9-9xnrj to ip-10-0-1-39.us-east-2.compute.internal 2026/07/12 18:59:01 18:54:24 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: Pulling - Pulling image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" 2026/07/12 18:59:01 18:54:24 INFO ReplicaSet/rhods-dashboard-5998574cd9: SuccessfulCreate - Created pod: rhods-dashboard-5998574cd9-9xnrj 2026/07/12 18:59:01 18:54:24 INFO Pod/rhods-dashboard-5998574cd9-9xnrj: AddedInterface - Add eth0 [10.130.10.248/23] from ovn-kubernetes 2026/07/12 18:59:01 18:54:24 INFO ReplicaSet/rhods-dashboard-5998574cd9: SuccessfulCreate - Created pod: rhods-dashboard-5998574cd9-vcwpc 2026/07/12 18:59:01 18:54:23 INFO Deployment/rhods-dashboard: ScalingReplicaSet - Scaled up replica set rhods-dashboard-5998574cd9 from 0 to 2 2026/07/12 18:59:01 18:54:11 INFO Lease/spark-operator-controller-lock: LeaderElection - spark-operator-controller-767899d44f-x4sf4_087472fd-e0e1-463a-b1ae-32e24c26cad5 became leader 2026/07/12 18:59:01 18:54:10 INFO Pod/spark-operator-webhook-9cb9bdf44-d85tl: Pulled - Container image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/12 18:59:01 18:54:10 INFO Pod/spark-operator-webhook-9cb9bdf44-d85tl: AddedInterface - Add eth0 [10.128.0.91/23] from ovn-kubernetes 2026/07/12 18:59:01 18:54:10 INFO Pod/spark-operator-webhook-9cb9bdf44-d85tl: Created - Created container: webhook 2026/07/12 18:59:01 18:54:10 INFO Pod/spark-operator-webhook-9cb9bdf44-d85tl: Started - Started container webhook 2026/07/12 18:59:01 18:54:09 INFO Pod/spark-operator-webhook-9cb9bdf44-d85tl: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-webhook-9cb9bdf44-d85tl to ip-10-0-1-120.us-east-2.compute.internal 2026/07/12 18:59:01 18:54:09 INFO Deployment/spark-operator-webhook: ScalingReplicaSet - Scaled up replica set spark-operator-webhook-9cb9bdf44 from 0 to 1 2026/07/12 18:59:01 18:54:09 INFO ReplicaSet/spark-operator-webhook-9cb9bdf44: SuccessfulCreate - Created pod: spark-operator-webhook-9cb9bdf44-d85tl 2026/07/12 18:59:01 18:54:09 INFO Pod/spark-operator-webhook-9cb9bdf44-jrgkb: Killing - Stopping container webhook 2026/07/12 18:59:01 18:54:09 INFO Pod/spark-operator-webhook-9cb9bdf44-j2fhf: Killing - Stopping container webhook 2026/07/12 18:59:01 === RESOURCE QUOTAS === 2026/07/12 18:59:01 No resource quotas found in monitored namespaces 2026/07/12 18:59:01 Classification for group 2: infrastructure/storage (code=1006, confidence=medium) 2026/07/12 18:59:01 Evidence: event Pod/odh-model-controller-867855946f-kh92j: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/odh-model-controller-867855946f-kh92j: FailedMount - MountVolume.SetUp failed for volume \"cert\" : secret \"odh-model-controller-webhook-cert\" not found"],"confidence":"medium"} 2026/07/12 18:59:01 Diagnostics complete for failed test: group 2 controller_test.go:753: Stopping: group 2 test failed. 2026/07/12 18:59:01 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/12 18:59:01 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/12 19:03:10 Circuit breaker: success recorded, resetting failure counter (was 1) 2026/07/12 19:13:20 TEST FAILURE DETECTED: group 4 2026/07/12 19:13:20 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 4) === 2026/07/12 19:13:21 === CLUSTER STATE === 2026/07/12 19:13:21 Node: ip-10-0-1-120.us-east-2.compute.internal 2026/07/12 19:13:21 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 19:13:21 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 19:13:21 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 19:13:21 Condition Ready: True - kubelet is posting ready status 2026/07/12 19:13:21 Resources: 7500m CPU, 31236228Ki memory allocatable (of 8000m CPU, 32387204Ki memory capacity) 2026/07/12 19:13:21 Node: ip-10-0-1-132.us-east-2.compute.internal 2026/07/12 19:13:21 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 19:13:21 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 19:13:21 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 19:13:21 Condition Ready: True - kubelet is posting ready status 2026/07/12 19:13:21 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/12 19:13:21 Node: ip-10-0-1-39.us-east-2.compute.internal 2026/07/12 19:13:21 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 19:13:21 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 19:13:21 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 19:13:21 Condition Ready: True - kubelet is posting ready status 2026/07/12 19:13:21 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/12 19:13:21 === DEPLOYMENTS === 2026/07/12 19:13:21 Failed to collect deployment data: deployments not ready: redhat-ods-applications/kserve-controller-manager (0/1) 2026/07/12 19:13:21 === PODS === 2026/07/12 19:13:21 Failed to collect pod data: redhat-ods-applications/kserve-controller-manager-6c67dcb69c-84lq8: phase=Pending; redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-qqqpv: container manager restarts=1; redhat-ods-applications/maas-controller-85bdcf9988-9lck9: container manager restarts=1 2026/07/12 19:13:21 === OPERATOR STATUS === 2026/07/12 19:13:21 Operator deployment: 3/3 ready 2026/07/12 19:13:21 === DSCI STATUS === 2026/07/12 19:13:21 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/12 19:13:21 === DSC STATUS === 2026/07/12 19:13:21 Failed to collect DSC data: default-dsc unhealthy conditions: Ready=False; ComponentsReady=False; KserveReady=False; MaaSPrerequisitesAvailable=False 2026/07/12 19:13:21 === RECENT EVENTS === 2026/07/12 19:13:21 19:12:52 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/12 19:13:21 19:09:10 INFO Pod/kserve-controller-manager-6c67dcb69c-84lq8: BackOff - Back-off pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/12 19:13:21 19:08:58 WARN Pod/kserve-controller-manager-6c67dcb69c-84lq8: Failed - Error: ImagePullBackOff 2026/07/12 19:13:21 === RESOURCE QUOTAS === 2026/07/12 19:13:21 No resource quotas found in monitored namespaces 2026/07/12 19:13:21 Classification for group 4: infrastructure/image-pull (code=1001, confidence=medium) 2026/07/12 19:13:21 Evidence: container kserve-controller-manager-6c67dcb69c-84lq8/manager waiting: ImagePullBackOff Back-off pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c: reading manifest sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c in registry.redhat.io/rhoai/odh-kserve-controller-rhel9: manifest unknown; artifact err: get manifest: build image source: reading manifest sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c in registry.redhat.io/rhoai/odh-kserve-controller-rhel9: manifest unknown FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"image-pull","error_code":1001,"evidence":["container kserve-controller-manager-6c67dcb69c-84lq8/manager waiting: ImagePullBackOff Back-off pulling image \"registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c\": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c: reading manifest sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c in registry.redhat.io/rhoai/odh-kserve-controller-rhel9: manifest unknown; artifact err: get manifest: build image source: reading manifest sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c in registry.redhat.io/rhoai/odh-kserve-controller-rhel9: manifest unknown"],"confidence":"medium"} 2026/07/12 19:13:21 Diagnostics complete for failed test: group 4 controller_test.go:753: Stopping: group 4 test failed. 2026/07/12 19:13:21 Circuit breaker: failure classified as infrastructure/image-pull (code=1001) 2026/07/12 19:13:21 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/12 19:14:22 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (1541.46s) === RUN TestOdhOperator 2026/07/12 18:49:31 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/12 19:15:23 TEST FAILURE DETECTED: components 2026/07/12 19:15:23 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/12 19:15:23 === CLUSTER STATE === 2026/07/12 19:15:23 Node: ip-10-0-1-120.us-east-2.compute.internal 2026/07/12 19:15:23 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 19:15:23 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 19:15:23 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 19:15:23 Condition Ready: True - kubelet is posting ready status 2026/07/12 19:15:23 Resources: 7500m CPU, 31236228Ki memory allocatable (of 8000m CPU, 32387204Ki memory capacity) 2026/07/12 19:15:23 Node: ip-10-0-1-132.us-east-2.compute.internal 2026/07/12 19:15:23 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 19:15:23 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 19:15:23 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 19:15:23 Condition Ready: True - kubelet is posting ready status 2026/07/12 19:15:23 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/12 19:15:23 Node: ip-10-0-1-39.us-east-2.compute.internal 2026/07/12 19:15:23 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 19:15:23 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 19:15:23 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 19:15:23 Condition Ready: True - kubelet is posting ready status 2026/07/12 19:15:23 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/12 19:15:23 === DEPLOYMENTS === 2026/07/12 19:15:23 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/12 19:15:23 === PODS === 2026/07/12 19:15:23 Failed to collect pod data: redhat-ods-applications/maas-controller-85bdcf9988-ckcwv: container manager not ready 2026/07/12 19:15:23 === OPERATOR STATUS === 2026/07/12 19:15:23 Operator deployment: 3/3 ready 2026/07/12 19:15:23 === DSCI STATUS === 2026/07/12 19:15:23 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/12 19:15:23 === DSC STATUS === 2026/07/12 19:15:23 Failed to collect DSC data: default-dsc unhealthy conditions: Ready=False; ComponentsReady=False; MaaSPrerequisitesAvailable=False; ModelsAsServiceReady=False 2026/07/12 19:15:23 === RECENT EVENTS === 2026/07/12 19:15:23 19:15:22 INFO Pod/maas-controller-85bdcf9988-ckcwv: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 583ms (583ms including waiting). Image size: 212866077 bytes. 2026/07/12 19:15:23 19:15:22 WARN Pod/maas-controller-85bdcf9988-ckcwv: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-zj2bh); attempting to pull the image may not succeed. 2026/07/12 19:15:23 19:15:22 INFO Pod/maas-controller-85bdcf9988-ckcwv: Started - Started container manager 2026/07/12 19:15:23 19:15:22 INFO Pod/maas-controller-85bdcf9988-ckcwv: Created - Created container: manager 2026/07/12 19:15:23 19:15:22 INFO Pod/maas-controller-85bdcf9988-ckcwv: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/12 19:15:23 19:15:08 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-q4pnf_4878a74c-13a6-43df-971f-047fd2f9cc4d became leader 2026/07/12 19:15:23 19:15:07 WARN Pod/maas-controller-85bdcf9988-ckcwv: BackOff - Back-off restarting failed container manager in pod maas-controller-85bdcf9988-ckcwv_redhat-ods-applications(92a0f5dc-c156-4623-8b1b-13d52a675c2d) 2026/07/12 19:15:23 19:15:05 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-2vljb_f38f5a03-7fa3-4cdc-999e-ec379bc863c9 became leader 2026/07/12 19:15:23 19:15:04 INFO CronJob/maas-api-key-cleanup: SawCompletedJob - Saw completed job: maas-api-key-cleanup-29731395, condition: Complete 2026/07/12 19:15:23 19:15:04 INFO Job/maas-api-key-cleanup-29731395: Completed - Job completed 2026/07/12 19:15:23 19:15:00 INFO Pod/maas-api-key-cleanup-29731395-zclgx: Scheduled - Successfully assigned redhat-ods-applications/maas-api-key-cleanup-29731395-zclgx to ip-10-0-1-120.us-east-2.compute.internal 2026/07/12 19:15:23 19:15:00 INFO CronJob/maas-api-key-cleanup: SuccessfulCreate - Created job maas-api-key-cleanup-29731395 2026/07/12 19:15:23 19:15:00 INFO Job/maas-api-key-cleanup-29731395: SuccessfulCreate - Created pod: maas-api-key-cleanup-29731395-zclgx 2026/07/12 19:15:23 19:15:00 INFO Pod/maas-api-key-cleanup-29731395-zclgx: Started - Started container cleanup 2026/07/12 19:15:23 19:15:00 INFO Pod/maas-api-key-cleanup-29731395-zclgx: Created - Created container: cleanup 2026/07/12 19:15:23 19:15:00 INFO Pod/maas-api-key-cleanup-29731395-zclgx: Pulled - Container image "registry.redhat.io/ubi9/ubi-minimal@sha256:907b68736aa798b2d38255b7aa070b2a70acb90803864a40f05d0ec47556ddd0" already present on machine 2026/07/12 19:15:23 19:15:00 INFO Pod/maas-api-key-cleanup-29731395-zclgx: AddedInterface - Add eth0 [10.128.0.110/23] from ovn-kubernetes 2026/07/12 19:15:23 19:14:53 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-vtrcb_816ae20c-d715-43f9-b75b-f183879158de became leader 2026/07/12 19:15:23 19:14:49 INFO Pod/llmisvc-controller-manager-5d65df97bb-q4pnf: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/12 19:15:23 19:14:49 INFO Pod/kserve-controller-manager-6c67dcb69c-2vljb: Started - Started container manager 2026/07/12 19:15:23 19:14:49 INFO Pod/llmisvc-controller-manager-5d65df97bb-q4pnf: Started - Started container manager 2026/07/12 19:15:23 19:14:49 INFO Pod/llmisvc-controller-manager-5d65df97bb-q4pnf: Created - Created container: manager 2026/07/12 19:15:23 19:14:49 INFO Pod/llmisvc-controller-manager-5d65df97bb-q4pnf: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 481ms (481ms including waiting). Image size: 236366599 bytes. 2026/07/12 19:15:23 19:14:49 INFO Pod/kserve-controller-manager-6c67dcb69c-2vljb: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/12 19:15:23 19:14:49 INFO Pod/llmisvc-controller-manager-5d65df97bb-q4pnf: AddedInterface - Add eth0 [10.130.11.33/23] from ovn-kubernetes 2026/07/12 19:15:23 19:14:49 INFO Pod/kserve-controller-manager-6c67dcb69c-2vljb: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 501ms (501ms including waiting). Image size: 238435613 bytes. 2026/07/12 19:15:23 19:14:49 INFO Pod/kserve-controller-manager-6c67dcb69c-2vljb: Created - Created container: manager 2026/07/12 19:15:23 19:14:48 INFO Pod/llmisvc-controller-manager-5d65df97bb-q4pnf: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-q4pnf to ip-10-0-1-39.us-east-2.compute.internal 2026/07/12 19:15:23 19:14:48 INFO Pod/kserve-controller-manager-6c67dcb69c-2vljb: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-2vljb to ip-10-0-1-39.us-east-2.compute.internal 2026/07/12 19:15:23 19:14:48 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-q4pnf 2026/07/12 19:15:23 19:14:48 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-2vljb 2026/07/12 19:15:23 19:14:48 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/12 19:15:23 19:14:48 INFO Pod/kserve-controller-manager-6c67dcb69c-2vljb: AddedInterface - Add eth0 [10.130.11.32/23] from ovn-kubernetes 2026/07/12 19:15:23 19:14:48 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/12 19:15:23 19:14:47 INFO Pod/maas-controller-85bdcf9988-ckcwv: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 478ms (478ms including waiting). Image size: 212866077 bytes. 2026/07/12 19:15:23 19:14:36 INFO Pod/model-serving-api-6688cd56bd-k8tq7: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-k8tq7 to ip-10-0-1-39.us-east-2.compute.internal 2026/07/12 19:15:23 19:14:36 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-k8tq7 2026/07/12 19:15:23 19:14:36 INFO Pod/model-serving-api-6688cd56bd-k8tq7: AddedInterface - Add eth0 [10.130.11.31/23] from ovn-kubernetes 2026/07/12 19:15:23 19:14:36 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/12 19:15:23 19:14:36 INFO Pod/maas-controller-85bdcf9988-ckcwv: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 524ms (524ms including waiting). Image size: 212866077 bytes. 2026/07/12 19:15:23 19:14:36 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/12 19:15:23 19:14:36 INFO Pod/model-serving-api-6688cd56bd-k8tq7: Started - Started container server 2026/07/12 19:15:23 19:14:36 INFO Pod/model-serving-api-6688cd56bd-k8tq7: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/12 19:15:23 19:14:36 INFO Pod/model-serving-api-6688cd56bd-k8tq7: Created - Created container: server 2026/07/12 19:15:23 19:14:35 INFO Pod/odh-model-controller-6849fc49d8-vtrcb: Created - Created container: manager 2026/07/12 19:15:23 19:14:35 INFO Pod/odh-model-controller-6849fc49d8-vtrcb: AddedInterface - Add eth0 [10.130.11.30/23] from ovn-kubernetes 2026/07/12 19:15:23 19:14:35 INFO Pod/odh-model-controller-6849fc49d8-vtrcb: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/12 19:15:23 19:14:35 INFO Pod/odh-model-controller-6849fc49d8-vtrcb: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 506ms (506ms including waiting). Image size: 238959091 bytes. 2026/07/12 19:15:23 19:14:35 INFO Pod/odh-model-controller-6849fc49d8-vtrcb: Started - Started container manager 2026/07/12 19:15:23 19:14:35 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/12 19:15:23 19:14:34 INFO Pod/odh-model-controller-6849fc49d8-vtrcb: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-vtrcb to ip-10-0-1-39.us-east-2.compute.internal 2026/07/12 19:15:23 19:14:34 WARN Pod/odh-model-controller-6849fc49d8-vtrcb: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/12 19:15:23 19:14:34 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-vtrcb 2026/07/12 19:15:23 19:14:33 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/12 19:15:23 19:14:24 INFO Pod/llmisvc-controller-manager-5d65df97bb-z46gk: Killing - Stopping container manager 2026/07/12 19:15:23 19:14:24 INFO Pod/model-serving-api-6688cd56bd-pqk8c: Killing - Stopping container server 2026/07/12 19:15:23 19:14:24 INFO Pod/odh-model-controller-6849fc49d8-qxjw4: Killing - Stopping container manager 2026/07/12 19:15:23 19:14:24 INFO Pod/kserve-controller-manager-6c67dcb69c-9sxsm: Killing - Stopping container manager 2026/07/12 19:15:23 19:14:20 INFO Pod/maas-api-84449bdd5d-nsks2: Started - Started container maas-api 2026/07/12 19:15:23 19:14:20 INFO Pod/maas-api-84449bdd5d-nsks2: Created - Created container: maas-api 2026/07/12 19:15:23 19:14:19 INFO Pod/maas-api-84449bdd5d-nsks2: Pulling - Pulling image "quay.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" 2026/07/12 19:15:23 19:14:19 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/12 19:15:23 19:14:19 INFO Pod/maas-api-84449bdd5d-nsks2: AddedInterface - Add eth0 [10.130.11.27/23] from ovn-kubernetes 2026/07/12 19:15:23 19:14:19 INFO Pod/maas-api-84449bdd5d-nsks2: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" in 565ms (565ms including waiting). Image size: 175018396 bytes. 2026/07/12 19:15:23 19:14:18 INFO Pod/maas-api-84449bdd5d-nsks2: Scheduled - Successfully assigned redhat-ods-applications/maas-api-84449bdd5d-nsks2 to ip-10-0-1-39.us-east-2.compute.internal 2026/07/12 19:15:23 19:14:18 WARN ReplicaSet/maas-api-84449bdd5d: FailedCreate - Error creating: pods "maas-api-84449bdd5d-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 2026/07/12 19:15:23 19:14:18 INFO Deployment/maas-api: ScalingReplicaSet - Scaled up replica set maas-api-84449bdd5d from 0 to 1 2026/07/12 19:15:23 19:14:18 INFO ReplicaSet/maas-api-84449bdd5d: SuccessfulCreate - Created pod: maas-api-84449bdd5d-nsks2 2026/07/12 19:15:23 19:14:01 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-9sxsm_29a4e89f-d15b-451d-8afe-b812338470e5 became leader 2026/07/12 19:15:23 19:14:00 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-z46gk_9931d64b-5908-4fcb-a359-c136704dfb1c became leader 2026/07/12 19:15:23 19:13:47 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-7ccfdd66-wsr4j_1072f938-d7e8-4917-9736-9a2530e2eb59 became leader 2026/07/12 19:15:23 19:13:47 INFO Lease/maas-controller.models-as-a-service.opendatahub.io: LeaderElection - maas-controller-85bdcf9988-ckcwv_9497b00b-9cf9-4513-addd-a60f1cd08a5a became leader 2026/07/12 19:15:23 19:13:44 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-qxjw4_b5d80fe7-a267-48cf-8b35-74e7721dfb1f became leader 2026/07/12 19:15:23 19:13:43 INFO Pod/kserve-controller-manager-6c67dcb69c-9sxsm: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 608ms (608ms including waiting). Image size: 238435613 bytes. 2026/07/12 19:15:23 19:13:43 INFO Pod/llmisvc-controller-manager-5d65df97bb-z46gk: Started - Started container manager 2026/07/12 19:15:23 19:13:43 INFO Pod/kserve-controller-manager-6c67dcb69c-9sxsm: Created - Created container: manager 2026/07/12 19:15:23 19:13:43 INFO Pod/kserve-controller-manager-6c67dcb69c-9sxsm: AddedInterface - Add eth0 [10.130.11.25/23] from ovn-kubernetes 2026/07/12 19:15:23 19:13:43 INFO Pod/kserve-controller-manager-6c67dcb69c-9sxsm: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/12 19:15:23 19:13:43 INFO Pod/kserve-controller-manager-6c67dcb69c-9sxsm: Started - Started container manager 2026/07/12 19:15:23 19:13:43 INFO Pod/llmisvc-controller-manager-5d65df97bb-z46gk: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 564ms (564ms including waiting). Image size: 236366599 bytes. 2026/07/12 19:15:23 19:13:43 INFO Pod/llmisvc-controller-manager-5d65df97bb-z46gk: Created - Created container: manager 2026/07/12 19:15:23 19:13:42 INFO Pod/llmisvc-controller-manager-5d65df97bb-z46gk: AddedInterface - Add eth0 [10.130.11.26/23] from ovn-kubernetes 2026/07/12 19:15:23 19:13:42 INFO Pod/llmisvc-controller-manager-5d65df97bb-z46gk: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/12 19:15:23 19:13:41 INFO Pod/llmisvc-controller-manager-5d65df97bb-z46gk: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-z46gk to ip-10-0-1-39.us-east-2.compute.internal 2026/07/12 19:15:23 19:13:41 INFO Pod/kserve-controller-manager-6c67dcb69c-9sxsm: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-9sxsm to ip-10-0-1-39.us-east-2.compute.internal 2026/07/12 19:15:23 19:13:41 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-9sxsm 2026/07/12 19:15:23 19:13:41 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/12 19:15:23 19:13:41 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-z46gk 2026/07/12 19:15:23 19:13:41 WARN Pod/kserve-controller-manager-6c67dcb69c-9sxsm: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 2026/07/12 19:15:23 19:13:41 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/12 19:15:23 19:13:29 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-wsr4j: Pulled - Container image "quay.io/rhoai/odh-trainer-rhel9@sha256:59ea523e2df9c1e705c755308ac11420682081cc63d54890e2b15fbd7d8ba570" already present on machine 2026/07/12 19:15:23 19:13:29 INFO Pod/model-serving-api-6688cd56bd-pqk8c: Created - Created container: server 2026/07/12 19:15:23 19:13:29 INFO Pod/model-serving-api-6688cd56bd-pqk8c: Started - Started container server 2026/07/12 19:15:23 19:13:29 INFO Pod/model-serving-api-6688cd56bd-pqk8c: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/12 19:15:23 19:13:29 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-wsr4j: Started - Started container manager 2026/07/12 19:15:23 19:13:29 INFO Pod/model-serving-api-6688cd56bd-pqk8c: AddedInterface - Add eth0 [10.130.11.24/23] from ovn-kubernetes 2026/07/12 19:15:23 19:13:29 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-wsr4j: AddedInterface - Add eth0 [10.128.0.109/23] from ovn-kubernetes 2026/07/12 19:15:23 19:13:29 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-wsr4j: Created - Created container: manager 2026/07/12 19:15:23 19:13:28 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-wsr4j: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-7ccfdd66-wsr4j to ip-10-0-1-120.us-east-2.compute.internal 2026/07/12 19:15:23 19:13:28 INFO Pod/model-serving-api-6688cd56bd-pqk8c: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-pqk8c to ip-10-0-1-39.us-east-2.compute.internal 2026/07/12 19:15:23 19:13:28 INFO Pod/odh-model-controller-6849fc49d8-qxjw4: Started - Started container manager 2026/07/12 19:15:23 19:13:28 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-pqk8c 2026/07/12 19:15:23 19:13:28 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/12 19:15:23 19:13:28 INFO Pod/odh-model-controller-6849fc49d8-qxjw4: AddedInterface - Add eth0 [10.130.11.22/23] from ovn-kubernetes 2026/07/12 19:15:23 19:13:28 INFO Pod/odh-model-controller-6849fc49d8-qxjw4: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/12 19:15:23 19:13:28 INFO Pod/odh-model-controller-6849fc49d8-qxjw4: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 522ms (522ms including waiting). Image size: 238959091 bytes. 2026/07/12 19:15:23 19:13:28 INFO Pod/odh-model-controller-6849fc49d8-qxjw4: Created - Created container: manager 2026/07/12 19:15:23 19:13:28 WARN Pod/model-serving-api-6688cd56bd-pqk8c: FailedMount - MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 2026/07/12 19:15:23 19:13:28 INFO Pod/maas-controller-85bdcf9988-ckcwv: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 432ms (432ms including waiting). Image size: 212866077 bytes. 2026/07/12 19:15:23 19:13:28 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-7ccfdd66 from 0 to 1 2026/07/12 19:15:23 19:13:28 INFO ReplicaSet/kubeflow-trainer-controller-manager-7ccfdd66: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-7ccfdd66-wsr4j 2026/07/12 19:15:23 19:13:28 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/12 19:15:23 19:13:27 INFO Pod/maas-controller-85bdcf9988-ckcwv: Scheduled - Successfully assigned redhat-ods-applications/maas-controller-85bdcf9988-ckcwv to ip-10-0-1-39.us-east-2.compute.internal 2026/07/12 19:15:23 19:13:27 INFO Pod/maas-controller-85bdcf9988-ckcwv: AddedInterface - Add eth0 [10.130.11.23/23] from ovn-kubernetes 2026/07/12 19:15:23 19:13:27 INFO ReplicaSet/maas-controller-85bdcf9988: SuccessfulCreate - Created pod: maas-controller-85bdcf9988-ckcwv 2026/07/12 19:15:23 19:13:26 INFO Pod/odh-model-controller-6849fc49d8-qxjw4: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-qxjw4 to ip-10-0-1-39.us-east-2.compute.internal 2026/07/12 19:15:23 19:13:26 INFO Deployment/maas-controller: ScalingReplicaSet - Scaled up replica set maas-controller-85bdcf9988 from 0 to 1 2026/07/12 19:15:23 19:13:26 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-qxjw4 2026/07/12 19:15:23 19:13:26 WARN Pod/odh-model-controller-6849fc49d8-qxjw4: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/12 19:15:23 19:13:25 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/12 19:15:23 19:13:23 INFO Pod/model-serving-api-6688cd56bd-g6brm: Killing - Stopping container server 2026/07/12 19:15:23 19:13:23 INFO Pod/odh-model-controller-6849fc49d8-j26rq: Killing - Stopping container manager 2026/07/12 19:15:23 19:13:23 INFO Pod/maas-controller-85bdcf9988-9lck9: Killing - Stopping container manager 2026/07/12 19:15:23 19:13:23 INFO Pod/maas-api-84449bdd5d-n58sh: Killing - Stopping container maas-api 2026/07/12 19:15:23 19:13:22 INFO Pod/llmisvc-controller-manager-5d65df97bb-qqqpv: Killing - Stopping container manager 2026/07/12 19:15:23 19:12:52 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/12 19:15:23 === RESOURCE QUOTAS === 2026/07/12 19:15:23 No resource quotas found in monitored namespaces 2026/07/12 19:15:23 Classification for components: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/12 19:15:23 Evidence: container maas-controller-85bdcf9988-ckcwv/manager waiting: CrashLoopBackOff back-off 20s restarting failed container=manager pod=maas-controller-85bdcf9988-ckcwv_redhat-ods-applications(92a0f5dc-c156-4623-8b1b-13d52a675c2d) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-85bdcf9988-ckcwv/manager waiting: CrashLoopBackOff back-off 20s restarting failed container=manager pod=maas-controller-85bdcf9988-ckcwv_redhat-ods-applications(92a0f5dc-c156-4623-8b1b-13d52a675c2d)"],"confidence":"medium"} 2026/07/12 19:15:23 Diagnostics complete for failed test: components controller_test.go:484: Stopping: components test failed. 2026/07/12 19:15:23 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/12 19:15:23 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/12 19:15:24 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (1553.95s)