=== RUN TestOdhOperator/components/group_2/kueue/Validate_component_removed_to_unmanaged_transition helper_test.go:150: Setting Kueue component (default-kueue) to Removed mode to start with a clean state. helper_test.go:150: Cleaning up any existing Kueue test resources. helper_test.go:150: Cleaning up Kueue cluster-scoped resources helper_test.go:150: Will attempt to delete 2 Kueue resources helper_test.go:150: Attempting to delete ClusterQueue /default helper_test.go:150: Successfully processed deletion of ClusterQueue /default helper_test.go:150: Attempting to delete Kueue /cluster helper_test.go:150: Successfully processed deletion of Kueue /cluster helper_test.go:150: Creating test namespace (test-kueue-managed-dbjll53j9n1c7l1nljeg) with Kueue management annotation. helper_test.go:150: Creating Kueue ConfigMap (kueue-manager-config) to verify default Kueue resource is created correctly. helper_test.go:150: Creating Kueue ConfigMap (kueue-manager-config) in namespace (redhat-ods-applications). helper_test.go:150: Updating the management state of the component in the DataScienceCluster to Unmanaged (expecting NotReady initially due to missing operator). helper_test.go:150: Ensuring LWS Operator is installed (namespace=openshift-lws-operator, name=leader-worker-set). helper_test.go:150: Ensuring LeaderWorkerSetOperator CR exists (namespace=openshift-lws-operator, name=cluster). helper_test.go:150: Ensuring OCP Kueue Operator is installed (namespace=openshift-kueue-operator, name=kueue-operator). helper_test.go:150: Verifying the component in the DataScienceCluster transitions to Ready state. helper_test.go:150: Verifying Kueue configuration (cluster) is created with all expected frameworks. testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_2/kueue/Validate_component_removed_to_unmanaged_transition (32.56s) === RUN TestOdhOperator/components/group_2/kueue === PAUSE TestOdhOperator/components/group_2/kueue === CONT TestOdhOperator/components/group_2/kueue test_context_test.go:223: Timed out after 10.001s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:114 with: Expected resource 'cluster' of kind 'Kueue' to match condition {(["BatchJob", "Deployment", "JobSet", "LeaderWorkerSet", "MPIJob", "PaddleJob", "Pod", "PyTorchJob", "RayCluster", "RayJob", "SparkApplication", "StatefulSet", "TFJob", "TrainJob", "XGBoostJob"] - .spec.config.integrations.frameworks) | length == 0 []}. Expected <string>: map[apiVersion:kueue.openshift.io/v1 kind:Kueue metadata:map[annotations:map[platform.opendatahub.io/instance.generation:1 platform.opendatahub.io/instance.name:default-kueue platform.opendatahub.io/instance.uid:9d81c209-4d61-4c87-8358-71b65007a060 platform.opendatahub.io/type:OpenShift AI Self-Managed platform.opendatahub.io/version:3.5.0-ea.2] creationTimestamp:2026-11-01T13:29:18Z deletionGracePeriodSeconds:0 deletionTimestamp:2026-11-01T15:28:31Z finalizers:[kueue.openshift.io/finalizer] generation:2 labels:map[platform.opendatahub.io/part-of:kueue] managedFields:[map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:status:map[f:conditions:map[k:{"type":"Available"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"CertManagerAvailable"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"Degraded"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"Progressing"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]]] f:readyReplicas:map[]]] manager:kueue-operator operation:Apply subresource:status time:2026-11-01T13:30:09Z] map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[.:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:labels:map[.:map[] f:platform.opendatahub.io/part-of:map[]]] f:spec:map[.:map[] f:config:map[.:map[] f:integrations:map[.:map[] f:frameworks:map[.:map[] v:"Deployment":map[] v:"Pod":map[] v:"PyTorchJob":map[] v:"RayCluster":map[] v:"RayJob":map[] v:"SparkApplication":map[] v:"StatefulSet":map[] v:"TrainJob":map[]]]] f:logLevel:map[] f:managementState:map[] f:operatorLogLevel:map[]]] manager:manager operation:Update time:2026-11-01T13:29:18Z] map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:finalizers:map[.:map[] v:"kueue.openshift.io/finalizer":map[]]]] manager:kueue-operator operation:Update time:2026-11-01T15:28:31Z]] name:cluster resourceVersion:45369148 uid:74b51c97-bc41-4b9c-a118-9a61de14a0c1] spec:map[config:map[integrations:map[frameworks:[Deployment Pod PyTorchJob RayCluster RayJob SparkApplication StatefulSet TrainJob]]] logLevel:Normal managementState:Managed operatorLogLevel:Normal] status:map[conditions:[map[lastTransitionTime:2026-11-01T13:30:09Z message:All 2 replicas are ready reason:AllReplicasReady status:True type:Available] map[lastTransitionTime:2026-11-01T13:30:09Z message:Deployment is up to date reason:AsExpected status:False type:Progressing] map[lastTransitionTime:2026-11-01T13:30:09Z message: reason:AsExpected status:False type:Degraded] map[lastTransitionTime:2026-11-01T13:29:50Z message:cert-manager is installed reason:CertManagerInstalled status:True type:CertManagerAvailable]] readyReplicas:2]] to match expression <string>: (["BatchJob", "Deployment", "JobSet", "LeaderWorkerSet", "MPIJob", "PaddleJob", "Pod", "PyTorchJob", "RayCluster", "RayJob", "SparkApplication", "StatefulSet", "TFJob", "TrainJob", "XGBoostJob"] - .spec.config.integrations.frameworks) | length == 0 test_context_test.go:1106: cleanup: best-effort delete failed for /, Kind= redhat-ods-applications/kueue-manager-config: Object 'Kind' is missing in 'unstructured object has no kind' --- FAIL: TestOdhOperator/components/group_2/kueue (33.34s) === RUN TestOdhOperator/components/group_2 --- FAIL: TestOdhOperator/components/group_2 (0.00s) === RUN TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable helper_test.go:150: Verifying Tenant models-as-a-service/default-tenant is present before disable helper_test.go:150: Disabling MaaS subcomponent (setting to Removed) helper_test.go:150: Waiting for Tenant models-as-a-service/default-tenant to be deleted testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable (430.64s) === RUN TestOdhOperator/components/group_4/modelsasservice === PAUSE TestOdhOperator/components/group_4/modelsasservice === CONT TestOdhOperator/components/group_4/modelsasservice 2026-11-01T15:36:27Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.22.15"}} controller_test.go:753: Creating MaaS PostgreSQL instance in namespace redhat-ods-applications controller_test.go:753: MaaS PostgreSQL instance and maas-db-config secret created in namespace redhat-ods-applications controller_test.go:753: Creating MaaS GatewayClass and Gateway: openshift-ingress/maas-default-gateway controller_test.go:753: Using hostname=maas.apps.rosa.konflux-rhoai1.84q6.s3.devshift.org, certSecret=2t60a0lj8jast7vfvf6elmkfit6temid-primary-cert-bundle-secret controller_test.go:753: MaaS Gateway openshift-ingress/maas-default-gateway created with HTTP+HTTPS listeners test_context_test.go:615: Timed out after 420.002s. The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1824 with: Expected resource list 'models-as-a-service/default-tenant' of kind 'Tenant' to be empty, but it contains resources. Expected <[]unstructured.Unstructured | len:1, cap:1>: [ { Object: { "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "kind": <string>"Tenant", "metadata": <map[string]interface {} | len:8>{ "creationTimestamp": <string>"2026-11-01T15:37:37Z", "generation": <int64>1, "managedFields": <[]interface {} | len:2, cap:2>[ <map[string]interface {} | len:6>{ "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:2>{ "f:metadata": <map[string]interface {} | len:1>{ "f:ownerReferences": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "k:{\"uid\":\"2d6a5ccc-e4b8-434b-883f-717497047aea\"}": <map[string]interface {} | len:0>{}, }, }, "f:spec": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "f:gatewayRef": <map[string]interface {} | len:3>{ "f:name": <map[string]interface {} | len:0>{}, "f:namespace": <map[string]interface {} | len:0>{}, ".": <map[string]interface {} | len:0>{}, }, }, }, "manager": <string>"manager", "operation": <string>"Update", "time": <string>"2026-11-01T15:37:38Z", }, <map[string]interface {} | len:7>{ "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:1>{ "f:status": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:conditions": <map[string]interface {} | len:0>{}, "f:phase": <map[string]interface {} | len:0>{}, }, }, "manager": <string>"manager", "operation": <string>"Update", "subresource": <string>"status", "time": <string>"2026-11-01T15:38:02Z", }, ], "name": <string>"default-tenant", "namespace": <string>"models-as-a-service", "ownerReferences": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:4>{ "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "kind": <string>"Config", "name": <string>"default", "uid": <string>"2d6a5ccc-e4b8-434b-883f-717497047aea", }, ], "resourceVersion": <string>"45382793", "uid": <string>"b3845027-9b09-4661-8e92-cc82384416cd", }, "spec": <map[string]interface {} | len:1>{ "gatewayRef": <map[string]interface {} | len:2>{ "name": <string>"maas-default-gateway", "namespace": <string>"openshift-ingress", }, }, "status": <map[string]interface {} | len:2>{ "conditions": <[]interface {} | len:5, cap:8>[ <map[string]interface {} | len:6>{ "observedGeneration": <int64>1, "reason": <string>"DependenciesMet", "status": <string>"True", "type": <string>"DependenciesAvailable", "lastTransitionTime": <string>"2026-11-01T15:37:40Z", "message": <string>"AuthConfig CRD (Kuadrant) is available on the cluster", }, <map[string]interface {} | len:6>{ "status": <string>"True", "type": <string>"MaaSPrerequisitesAvailable", "lastTransitionTime": <string>"2026-11-01T15:37:40Z", "message": <string>"All prerequisites are satisfied", "observedGeneration": <int64>1, "reason": <string>"PrerequisitesMet", }, <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-11-01T15:37:40Z", "message": <string>"All prerequisites are satisfied", "observedGeneration": <int64>1, "reason": <string>"PrerequisitesMet", "status": <string>"False", "type": <string>"Degraded", }, <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-11-01T15:38:02Z", "message": <string>"maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"DeploymentsReady", "status": <string>"True", "type": <string>"DeploymentsAvailable", }, <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-11-01T15:38:02Z", "message": <string>"MaaS platform manifests applied and maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"Reconciled", "status": <string>"True", "type": <string>"Ready", }, ], "phase": <string>"Active", }, }, }, ] to be empty --- FAIL: TestOdhOperator/components/group_4/modelsasservice (535.33s) === RUN TestOdhOperator/components/group_4 --- FAIL: TestOdhOperator/components/group_4 (0.00s) === RUN TestOdhOperator/components 2026/11/01 15:32:07 TEST FAILURE DETECTED: group 2 2026/11/01 15:32:07 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 2) === 2026/11/01 15:32:09 === CLUSTER STATE === 2026/11/01 15:32:09 Node: ip-10-0-1-134.us-west-2.compute.internal 2026/11/01 15:32:09 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/11/01 15:32:09 Condition DiskPressure: False - kubelet has no disk pressure 2026/11/01 15:32:09 Condition PIDPressure: False - kubelet has sufficient PID available 2026/11/01 15:32:09 Condition Ready: True - kubelet is posting ready status 2026/11/01 15:32:09 Resources: 15500m CPU, 63870868Ki memory allocatable (of 16000m CPU, 65021844Ki memory capacity) 2026/11/01 15:32:09 Node: ip-10-0-1-150.us-west-2.compute.internal 2026/11/01 15:32:09 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/11/01 15:32:09 Condition DiskPressure: False - kubelet has no disk pressure 2026/11/01 15:32:09 Condition PIDPressure: False - kubelet has sufficient PID available 2026/11/01 15:32:09 Condition Ready: True - kubelet is posting ready status 2026/11/01 15:32:09 Resources: 15500m CPU, 63870872Ki memory allocatable (of 16000m CPU, 65021848Ki memory capacity) 2026/11/01 15:32:09 Node: ip-10-0-1-85.us-west-2.compute.internal 2026/11/01 15:32:09 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/11/01 15:32:09 Condition DiskPressure: False - kubelet has no disk pressure 2026/11/01 15:32:09 Condition PIDPressure: False - kubelet has sufficient PID available 2026/11/01 15:32:09 Condition Ready: True - kubelet is posting ready status 2026/11/01 15:32:09 Resources: 15500m CPU, 63870868Ki memory allocatable (of 16000m CPU, 65021844Ki memory capacity) 2026/11/01 15:32:09 === DEPLOYMENTS === 2026/11/01 15:32:09 Namespace: redhat-ods-applications 2026/11/01 15:32:09 Deployment maas-api: 1/1 (READY) 2026/11/01 15:32:09 Deployment maas-controller: 1/1 (READY) 2026/11/01 15:32:09 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/11/01 15:32:09 Deployment spark-operator-controller: 1/1 (READY) 2026/11/01 15:32:09 Deployment spark-operator-webhook: 1/1 (READY) 2026/11/01 15:32:09 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/11/01 15:32:09 Namespace: kube-system 2026/11/01 15:32:09 Deployment kas-connection-checker: 3/3 (READY) 2026/11/01 15:32:09 === PODS === 2026/11/01 15:32:09 Namespace kube-system: no problematic pods found 2026/11/01 15:32:09 Namespace: redhat-ods-applications 2026/11/01 15:32:09 Pod maas-api-key-cleanup-29892450-jjzwl: Succeeded 2026/11/01 15:32:09 Container cleanup: ready=false, terminated: Completed (exit 0) 2026/11/01 15:32:09 === LOGS (CURRENT) for container cleanup === 2026/11/01 15:32:09 {"deletedCount":0,"message":"Successfully deleted 0 expired ephemeral key(s)"} 2026/11/01 15:32:09 === END LOGS === 2026/11/01 15:32:09 === OPERATOR STATUS === 2026/11/01 15:32:09 Operator deployment: 3/3 ready 2026/11/01 15:32:09 === DSCI STATUS === 2026/11/01 15:32:09 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/11/01 15:32:09 === DSC STATUS === 2026/11/01 15:32:09 Failed to collect DSC data: default-dsc unhealthy conditions: ModelsAsServiceReady=False 2026/11/01 15:32:09 === RECENT EVENTS === 2026/11/01 15:32:09 15:32:01 INFO Pod/odh-model-controller-5dbc549558-ng9jz: Killing - Stopping container manager 2026/11/01 15:32:09 15:32:01 INFO Pod/llmisvc-controller-manager-78d4c86675-frkht: Killing - Stopping container manager 2026/11/01 15:32:09 15:32:01 INFO Pod/model-registry-operator-controller-manager-54dd9ff76f-9kfhq: Killing - Stopping container manager 2026/11/01 15:32:09 15:32:01 INFO Pod/workload-variant-autoscaler-controller-manager-54bc958497-5b7kw: Killing - Stopping container manager 2026/11/01 15:32:09 15:32:01 INFO Pod/model-serving-api-5f9bc6947d-l7cwt: Killing - Stopping container server 2026/11/01 15:32:09 15:32:01 INFO Pod/kserve-controller-manager-8558c8ff56-zftzl: Killing - Stopping container manager 2026/11/01 15:32:09 15:31:45 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-5dbc549558-ng9jz_53b7402a-fdb3-48e2-9add-d998508833ac became leader 2026/11/01 15:32:09 15:31:42 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-54bc958497-5b7kw_d034e880-5874-4655-ae85-dac8d3a75095 became leader 2026/11/01 15:32:09 15:31:42 INFO Pod/workload-variant-autoscaler-controller-manager-54bc958497-5b7kw: AddedInterface - Add eth0 [10.128.0.38/23] from ovn-kubernetes 2026/11/01 15:32:09 15:31:42 INFO Pod/workload-variant-autoscaler-controller-manager-54bc958497-5b7kw: Pulled - Container image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" already present on machine and can be accessed by the pod 2026/11/01 15:32:09 15:31:42 INFO Pod/workload-variant-autoscaler-controller-manager-54bc958497-5b7kw: Created - Container created 2026/11/01 15:32:09 15:31:42 INFO Pod/workload-variant-autoscaler-controller-manager-54bc958497-5b7kw: Started - Container started 2026/11/01 15:32:09 15:31:42 INFO Pod/workload-variant-autoscaler-controller-manager-54bc958497-n9mhd: Killing - Stopping container manager 2026/11/01 15:32:09 15:31:41 INFO Pod/workload-variant-autoscaler-controller-manager-54bc958497-5b7kw: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-54bc958497-5b7kw to ip-10-0-1-150.us-west-2.compute.internal 2026/11/01 15:32:09 15:31:41 INFO Pod/workload-variant-autoscaler-controller-manager-54bc958497-mwhxx: Killing - Stopping container manager 2026/11/01 15:32:09 15:31:41 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-54bc958497 from 0 to 1 2026/11/01 15:32:09 15:31:41 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-54bc958497: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-54bc958497-5b7kw 2026/11/01 15:32:09 15:31:27 INFO Pod/odh-model-controller-5dbc549558-ng9jz: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 1.012s (1.012s including waiting). Image size: 238959091 bytes. 2026/11/01 15:32:09 15:31:27 INFO Pod/odh-model-controller-5dbc549558-ng9jz: Created - Container created 2026/11/01 15:32:09 15:31:27 INFO Pod/odh-model-controller-5dbc549558-ng9jz: Started - Container started 2026/11/01 15:32:09 15:31:26 INFO Pod/odh-model-controller-5dbc549558-ng9jz: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-5dbc549558-ng9jz to ip-10-0-1-150.us-west-2.compute.internal 2026/11/01 15:32:09 15:31:26 INFO Pod/odh-model-controller-5dbc549558-vhrfh: Killing - Stopping container manager 2026/11/01 15:32:09 15:31:26 INFO ReplicaSet/odh-model-controller-5dbc549558: SuccessfulCreate - Created pod: odh-model-controller-5dbc549558-ng9jz 2026/11/01 15:32:09 15:31:26 INFO Pod/odh-model-controller-5dbc549558-lsh4w: Killing - Stopping container manager 2026/11/01 15:32:09 15:31:26 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-5dbc549558 from 0 to 1 2026/11/01 15:32:09 15:31:26 INFO Pod/odh-model-controller-5dbc549558-ng9jz: AddedInterface - Add eth0 [10.128.0.37/23] from ovn-kubernetes 2026/11/01 15:32:09 15:31:26 INFO Pod/odh-model-controller-5dbc549558-ng9jz: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/11/01 15:32:09 15:31:11 INFO Pod/model-serving-api-5f9bc6947d-l7cwt: Created - Container created 2026/11/01 15:32:09 15:31:11 INFO Pod/model-serving-api-5f9bc6947d-l7cwt: Started - Container started 2026/11/01 15:32:09 15:31:11 INFO Pod/model-serving-api-5f9bc6947d-l7cwt: AddedInterface - Add eth0 [10.128.0.36/23] from ovn-kubernetes 2026/11/01 15:32:09 15:31:11 INFO Pod/model-serving-api-5f9bc6947d-l7cwt: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine and can be accessed by the pod 2026/11/01 15:32:09 15:31:10 INFO Pod/model-serving-api-5f9bc6947d-l7cwt: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5f9bc6947d-l7cwt to ip-10-0-1-150.us-west-2.compute.internal 2026/11/01 15:32:09 15:31:10 INFO ReplicaSet/model-serving-api-5f9bc6947d: SuccessfulCreate - Created pod: model-serving-api-5f9bc6947d-l7cwt 2026/11/01 15:32:09 15:31:10 INFO Pod/model-serving-api-5f9bc6947d-c9jpc: Killing - Stopping container server 2026/11/01 15:32:09 15:31:10 INFO Pod/model-serving-api-5f9bc6947d-hs6k7: Killing - Stopping container server 2026/11/01 15:32:09 15:31:10 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5f9bc6947d from 0 to 1 2026/11/01 15:32:09 15:30:08 INFO Pod/workload-variant-autoscaler-controller-manager-54bc958497-n9mhd: Created - Container created 2026/11/01 15:32:09 15:30:08 INFO Pod/workload-variant-autoscaler-controller-manager-54bc958497-n9mhd: AddedInterface - Add eth0 [10.129.0.171/23] from ovn-kubernetes 2026/11/01 15:32:09 15:30:08 INFO Pod/workload-variant-autoscaler-controller-manager-54bc958497-n9mhd: Pulled - Container image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" already present on machine and can be accessed by the pod 2026/11/01 15:32:09 15:30:08 INFO Pod/workload-variant-autoscaler-controller-manager-54bc958497-n9mhd: Started - Container started 2026/11/01 15:32:09 15:30:07 INFO Pod/workload-variant-autoscaler-controller-manager-54bc958497-n9mhd: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-54bc958497-n9mhd to ip-10-0-1-85.us-west-2.compute.internal 2026/11/01 15:32:09 15:30:07 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-54bc958497 from 1 to 2 2026/11/01 15:32:09 15:30:07 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-54bc958497: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-54bc958497-n9mhd 2026/11/01 15:32:09 15:30:04 INFO Job/maas-api-key-cleanup-29892450: Completed - Job completed 2026/11/01 15:32:09 15:30:04 INFO CronJob/maas-api-key-cleanup: SawCompletedJob - Saw completed job: maas-api-key-cleanup-29892450, condition: Complete 2026/11/01 15:32:09 15:30:01 INFO Pod/maas-api-key-cleanup-29892450-jjzwl: Started - Container started 2026/11/01 15:32:09 15:30:01 INFO Pod/maas-api-key-cleanup-29892450-jjzwl: Created - Container created 2026/11/01 15:32:09 15:30:00 INFO Pod/maas-api-key-cleanup-29892450-jjzwl: Scheduled - Successfully assigned redhat-ods-applications/maas-api-key-cleanup-29892450-jjzwl to ip-10-0-1-150.us-west-2.compute.internal 2026/11/01 15:32:09 15:30:00 INFO Pod/maas-api-key-cleanup-29892450-jjzwl: AddedInterface - Add eth0 [10.128.0.34/23] from ovn-kubernetes 2026/11/01 15:32:09 15:30:00 INFO Pod/maas-api-key-cleanup-29892450-jjzwl: Pulled - Container image "registry.redhat.io/ubi9/ubi-minimal@sha256:907b68736aa798b2d38255b7aa070b2a70acb90803864a40f05d0ec47556ddd0" already present on machine and can be accessed by the pod 2026/11/01 15:32:09 15:30:00 INFO Job/maas-api-key-cleanup-29892450: SuccessfulCreate - Created pod: maas-api-key-cleanup-29892450-jjzwl 2026/11/01 15:32:09 15:30:00 INFO CronJob/maas-api-key-cleanup: SuccessfulCreate - Created job maas-api-key-cleanup-29892450 2026/11/01 15:32:09 15:29:49 INFO Pod/odh-model-controller-5dbc549558-vhrfh: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 866ms (866ms including waiting). Image size: 238959091 bytes. 2026/11/01 15:32:09 15:29:49 INFO Pod/odh-model-controller-5dbc549558-vhrfh: Created - Container created 2026/11/01 15:32:09 15:29:49 INFO Pod/odh-model-controller-5dbc549558-vhrfh: Started - Container started 2026/11/01 15:32:09 15:29:48 INFO Pod/odh-model-controller-5dbc549558-vhrfh: AddedInterface - Add eth0 [10.129.0.170/23] from ovn-kubernetes 2026/11/01 15:32:09 15:29:48 INFO Pod/odh-model-controller-5dbc549558-vhrfh: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/11/01 15:32:09 15:29:47 INFO Pod/odh-model-controller-5dbc549558-vhrfh: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-5dbc549558-vhrfh to ip-10-0-1-85.us-west-2.compute.internal 2026/11/01 15:32:09 15:29:47 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-5dbc549558 from 1 to 2 2026/11/01 15:32:09 15:29:47 INFO ReplicaSet/odh-model-controller-5dbc549558: SuccessfulCreate - Created pod: odh-model-controller-5dbc549558-vhrfh 2026/11/01 15:32:09 15:29:28 INFO Pod/model-serving-api-5f9bc6947d-hs6k7: Started - Container started 2026/11/01 15:32:09 15:29:28 INFO Pod/model-serving-api-5f9bc6947d-hs6k7: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine and can be accessed by the pod 2026/11/01 15:32:09 15:29:28 INFO Pod/model-serving-api-5f9bc6947d-hs6k7: Created - Container created 2026/11/01 15:32:09 15:29:28 INFO Pod/model-serving-api-5f9bc6947d-hs6k7: AddedInterface - Add eth0 [10.129.0.169/23] from ovn-kubernetes 2026/11/01 15:32:09 15:29:27 INFO Pod/model-serving-api-5f9bc6947d-hs6k7: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5f9bc6947d-hs6k7 to ip-10-0-1-85.us-west-2.compute.internal 2026/11/01 15:32:09 15:29:27 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5f9bc6947d from 1 to 2 2026/11/01 15:32:09 15:29:27 INFO ReplicaSet/model-serving-api-5f9bc6947d: SuccessfulCreate - Created pod: model-serving-api-5f9bc6947d-hs6k7 2026/11/01 15:32:09 15:29:13 INFO Pod/maas-api-759b8d4d45-2jwhj: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" in 1.135s (1.135s including waiting). Image size: 175018396 bytes. 2026/11/01 15:32:09 15:29:13 INFO Pod/maas-api-759b8d4d45-2jwhj: Created - Container created 2026/11/01 15:32:09 15:29:13 INFO Pod/maas-api-759b8d4d45-2jwhj: Started - Container started 2026/11/01 15:32:09 15:29:11 INFO Pod/maas-api-759b8d4d45-2jwhj: Scheduled - Successfully assigned redhat-ods-applications/maas-api-759b8d4d45-2jwhj to ip-10-0-1-85.us-west-2.compute.internal 2026/11/01 15:32:09 15:29:11 INFO ReplicaSet/maas-api-759b8d4d45: SuccessfulCreate - Created pod: maas-api-759b8d4d45-2jwhj 2026/11/01 15:32:09 15:29:11 WARN ReplicaSet/maas-api-759b8d4d45: FailedCreate - Error creating: pods "maas-api-759b8d4d45-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 2026/11/01 15:32:09 15:29:11 INFO Pod/maas-api-759b8d4d45-2jwhj: Pulling - Pulling image "quay.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" 2026/11/01 15:32:09 15:29:11 INFO Pod/maas-api-759b8d4d45-2jwhj: AddedInterface - Add eth0 [10.129.0.164/23] from ovn-kubernetes 2026/11/01 15:32:09 15:29:11 INFO Deployment/maas-api: ScalingReplicaSet - Scaled up replica set maas-api-759b8d4d45 from 0 to 1 2026/11/01 15:32:09 15:28:59 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-78d4c86675-frkht_3a7ebe4f-c280-4d89-8535-4919b194a9f0 became leader 2026/11/01 15:32:09 15:28:57 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-8558c8ff56-zftzl_46088255-c692-45a4-97bd-cbc763c5e255 became leader 2026/11/01 15:32:09 15:28:47 INFO Pod/workload-variant-autoscaler-controller-manager-54bc958497-mwhxx: Created - Container created 2026/11/01 15:32:09 15:28:47 INFO Pod/workload-variant-autoscaler-controller-manager-54bc958497-mwhxx: Pulled - Container image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" already present on machine and can be accessed by the pod 2026/11/01 15:32:09 15:28:47 INFO Pod/workload-variant-autoscaler-controller-manager-54bc958497-mwhxx: Started - Container started 2026/11/01 15:32:09 15:28:47 INFO Pod/workload-variant-autoscaler-controller-manager-54bc958497-mwhxx: AddedInterface - Add eth0 [10.128.0.32/23] from ovn-kubernetes 2026/11/01 15:32:09 15:28:47 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-54bc958497-mwhxx_8fb8abad-8ee9-4083-9273-51b68c56210a became leader 2026/11/01 15:32:09 15:28:46 WARN ServiceMonitor/workload-variant-autoscaler-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/workload-variant-autoscaler-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: bearerTokenSecret: [REDACTED] to get secret "workload-variant-autoscaler-controller-manager-token": secrets "workload-variant-autoscaler-controller-manager-token" not found 2026/11/01 15:32:09 15:28:46 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/11/01 15:32:09 15:28:45 INFO Pod/workload-variant-autoscaler-controller-manager-54bc958497-mwhxx: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-54bc958497-mwhxx to ip-10-0-1-150.us-west-2.compute.internal 2026/11/01 15:32:09 15:28:45 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-54bc958497: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-54bc958497-mwhxx 2026/11/01 15:32:09 15:28:45 WARN Pod/workload-variant-autoscaler-controller-manager-54bc958497-mwhxx: FailedMount - MountVolume.SetUp failed for volume "epp-metrics-token" : secret "workload-variant-autoscaler-epp-metrics-token" not found 2026/11/01 15:32:09 15:28:44 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-54bc958497 from 0 to 1 2026/11/01 15:32:09 15:28:44 WARN ReplicaSet/workload-variant-autoscaler-controller-manager-54bc958497: FailedCreate - Error creating: pods "workload-variant-autoscaler-controller-manager-54bc958497-" is forbidden: error looking up service account redhat-ods-applications/workload-variant-autoscaler-controller-manager: serviceaccount "workload-variant-autoscaler-controller-manager" not found 2026/11/01 15:32:09 15:28:43 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-5dbc549558-lsh4w_84bd9428-173f-45ba-b4c2-80e248e86eca became leader 2026/11/01 15:32:09 15:28:42 INFO Pod/llmisvc-controller-manager-78d4c86675-frkht: Started - Container started 2026/11/01 15:32:09 15:28:42 INFO Pod/llmisvc-controller-manager-78d4c86675-frkht: Created - Container created 2026/11/01 15:32:09 15:28:41 INFO Pod/kserve-controller-manager-8558c8ff56-zftzl: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 978ms (978ms including waiting). Image size: 238435613 bytes. 2026/11/01 15:32:09 15:28:41 INFO Pod/kserve-controller-manager-8558c8ff56-zftzl: Started - Container started 2026/11/01 15:32:09 15:28:41 INFO Pod/kserve-controller-manager-8558c8ff56-zftzl: Created - Container created 2026/11/01 15:32:09 15:28:41 INFO Pod/llmisvc-controller-manager-78d4c86675-frkht: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 1.087s (1.087s including waiting). Image size: 236366599 bytes. 2026/11/01 15:32:09 15:28:40 INFO Pod/llmisvc-controller-manager-78d4c86675-frkht: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/11/01 15:32:09 15:28:40 INFO Pod/kserve-controller-manager-8558c8ff56-zftzl: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/11/01 15:32:09 15:28:40 INFO Lease/85f368d1.opendatahub.io: LeaderElection - model-registry-operator-controller-manager-54dd9ff76f-9kfhq_461c75d4-a8fd-4f37-a532-086d578bd0ae became leader 2026/11/01 15:32:09 15:28:40 INFO Pod/llmisvc-controller-manager-78d4c86675-frkht: AddedInterface - Add eth0 [10.128.0.24/23] from ovn-kubernetes 2026/11/01 15:32:09 15:28:40 INFO Pod/kserve-controller-manager-8558c8ff56-zftzl: AddedInterface - Add eth0 [10.128.0.15/23] from ovn-kubernetes 2026/11/01 15:32:09 15:28:39 INFO Pod/llmisvc-controller-manager-78d4c86675-frkht: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-78d4c86675-frkht to ip-10-0-1-150.us-west-2.compute.internal 2026/11/01 15:32:09 15:28:39 INFO Pod/kserve-controller-manager-8558c8ff56-zftzl: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-8558c8ff56-zftzl to ip-10-0-1-150.us-west-2.compute.internal 2026/11/01 15:32:09 15:28:39 INFO ReplicaSet/kserve-controller-manager-8558c8ff56: SuccessfulCreate - Created pod: kserve-controller-manager-8558c8ff56-zftzl 2026/11/01 15:32:09 15:28:39 INFO Lease/maas-controller.models-as-a-service.opendatahub.io: LeaderElection - maas-controller-86f7886655-5f4zp_8d1dbf3b-1f5f-422a-954e-de9090ea6f43 became leader 2026/11/01 15:32:09 15:28:39 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-8558c8ff56 from 0 to 1 2026/11/01 15:32:09 15:28:39 INFO ReplicaSet/llmisvc-controller-manager-78d4c86675: SuccessfulCreate - Created pod: llmisvc-controller-manager-78d4c86675-frkht 2026/11/01 15:32:09 15:28:39 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-78d4c86675 from 0 to 1 2026/11/01 15:32:09 15:28:26 INFO Pod/model-serving-api-5f9bc6947d-c9jpc: Started - Container started 2026/11/01 15:32:09 15:28:26 INFO Pod/model-serving-api-5f9bc6947d-c9jpc: AddedInterface - Add eth0 [10.128.0.7/23] from ovn-kubernetes 2026/11/01 15:32:09 15:28:26 INFO Pod/model-serving-api-5f9bc6947d-c9jpc: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine and can be accessed by the pod 2026/11/01 15:32:09 15:28:26 INFO Pod/model-serving-api-5f9bc6947d-c9jpc: Created - Container created 2026/11/01 15:32:09 15:28:25 INFO Pod/model-serving-api-5f9bc6947d-c9jpc: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5f9bc6947d-c9jpc to ip-10-0-1-150.us-west-2.compute.internal 2026/11/01 15:32:09 15:28:25 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5f9bc6947d from 0 to 1 2026/11/01 15:32:09 15:28:25 INFO Pod/odh-model-controller-5dbc549558-lsh4w: Created - Container created 2026/11/01 15:32:09 15:28:25 INFO Pod/odh-model-controller-5dbc549558-lsh4w: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 1.022s (1.022s including waiting). Image size: 238959091 bytes. 2026/11/01 15:32:09 15:28:25 WARN ReplicaSet/model-serving-api-5f9bc6947d: FailedCreate - Error creating: pods "model-serving-api-5f9bc6947d-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/11/01 15:32:09 15:28:25 INFO ReplicaSet/model-serving-api-5f9bc6947d: SuccessfulCreate - Created pod: model-serving-api-5f9bc6947d-c9jpc 2026/11/01 15:32:09 15:28:25 INFO Pod/odh-model-controller-5dbc549558-lsh4w: Started - Container started 2026/11/01 15:32:09 15:28:24 INFO Pod/maas-controller-86f7886655-5f4zp: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 943ms (943ms including waiting). Image size: 212866077 bytes. 2026/11/01 15:32:09 15:28:24 INFO Pod/maas-controller-86f7886655-5f4zp: Started - Container started 2026/11/01 15:32:09 15:28:24 INFO Pod/odh-model-controller-5dbc549558-lsh4w: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/11/01 15:32:09 15:28:24 INFO Pod/odh-model-controller-5dbc549558-lsh4w: AddedInterface - Add eth0 [10.128.0.3/23] from ovn-kubernetes 2026/11/01 15:32:09 15:28:24 INFO Pod/maas-controller-86f7886655-5f4zp: Created - Container created 2026/11/01 15:32:09 15:28:23 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/11/01 15:32:09 15:28:23 INFO Pod/model-registry-operator-controller-manager-54dd9ff76f-9kfhq: Created - Container created 2026/11/01 15:32:09 15:28:23 INFO Pod/model-registry-operator-controller-manager-54dd9ff76f-9kfhq: AddedInterface - Add eth0 [10.128.1.253/23] from ovn-kubernetes 2026/11/01 15:32:09 15:28:23 INFO Pod/model-registry-operator-controller-manager-54dd9ff76f-9kfhq: Pulled - Container image "quay.io/rhoai/odh-model-registry-operator-rhel9@sha256:ebcec5c43468283ed3cbab50530f939166f8f3e8c9d306e91a8107d05d7fa9d4" already present on machine and can be accessed by the pod 2026/11/01 15:32:09 15:28:23 WARN Pod/odh-model-controller-5dbc549558-lsh4w: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/11/01 15:32:09 15:28:23 INFO Pod/model-registry-operator-controller-manager-54dd9ff76f-9kfhq: Started - Container started 2026/11/01 15:32:09 15:28:23 INFO Pod/maas-controller-86f7886655-5f4zp: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/11/01 15:32:09 15:28:23 INFO Pod/maas-controller-86f7886655-5f4zp: AddedInterface - Add eth0 [10.128.1.254/23] from ovn-kubernetes 2026/11/01 15:32:09 15:28:22 INFO Pod/odh-model-controller-5dbc549558-lsh4w: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-5dbc549558-lsh4w to ip-10-0-1-150.us-west-2.compute.internal 2026/11/01 15:32:09 15:28:22 INFO Pod/maas-controller-86f7886655-5f4zp: Scheduled - Successfully assigned redhat-ods-applications/maas-controller-86f7886655-5f4zp to ip-10-0-1-150.us-west-2.compute.internal 2026/11/01 15:32:09 15:28:22 INFO Pod/model-registry-operator-controller-manager-54dd9ff76f-9kfhq: Scheduled - Successfully assigned redhat-ods-applications/model-registry-operator-controller-manager-54dd9ff76f-9kfhq to ip-10-0-1-150.us-west-2.compute.internal 2026/11/01 15:32:09 15:28:22 INFO Deployment/maas-controller: ScalingReplicaSet - Scaled up replica set maas-controller-86f7886655 from 0 to 1 2026/11/01 15:32:09 15:28:22 INFO ReplicaSet/odh-model-controller-5dbc549558: SuccessfulCreate - Created pod: odh-model-controller-5dbc549558-lsh4w 2026/11/01 15:32:09 15:28:22 INFO Deployment/model-registry-operator-controller-manager: ScalingReplicaSet - Scaled up replica set model-registry-operator-controller-manager-54dd9ff76f from 0 to 1 2026/11/01 15:32:09 15:28:22 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-5dbc549558 from 0 to 1 2026/11/01 15:32:09 15:28:22 INFO ReplicaSet/model-registry-operator-controller-manager-54dd9ff76f: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-54dd9ff76f-9kfhq 2026/11/01 15:32:09 15:28:22 WARN Pod/model-registry-operator-controller-manager-54dd9ff76f-9kfhq: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 2026/11/01 15:32:09 15:28:22 INFO ReplicaSet/maas-controller-86f7886655: SuccessfulCreate - Created pod: maas-controller-86f7886655-5f4zp 2026/11/01 15:32:09 15:28:12 INFO Pod/dashboard-redirect-7b8ddbf77d-9q47s: Killing - Stopping container nginx 2026/11/01 15:32:09 15:28:12 INFO Pod/dashboard-redirect-7b8ddbf77d-vjv29: Killing - Stopping container nginx 2026/11/01 15:32:09 15:27:31 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" in 948ms (948ms including waiting). Image size: 185092464 bytes. 2026/11/01 15:32:09 15:27:31 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" in 985ms (985ms including waiting). Image size: 185092464 bytes. 2026/11/01 15:32:09 15:27:30 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: Created - Container created 2026/11/01 15:32:09 15:27:30 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" 2026/11/01 15:32:09 15:27:30 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: Started - Container started 2026/11/01 15:32:09 15:27:30 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: Created - Container created 2026/11/01 15:32:09 15:27:30 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" in 933ms (933ms including waiting). Image size: 177808591 bytes. 2026/11/01 15:32:09 15:27:30 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: Started - Container started 2026/11/01 15:32:09 15:27:30 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" in 923ms (923ms including waiting). Image size: 177808591 bytes. 2026/11/01 15:32:09 15:27:30 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" 2026/11/01 15:32:09 15:27:29 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: Started - Container started 2026/11/01 15:32:09 15:27:29 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: Started - Container started 2026/11/01 15:32:09 15:27:29 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" in 977ms (977ms including waiting). Image size: 187802205 bytes. 2026/11/01 15:32:09 15:27:29 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: Created - Container created 2026/11/01 15:32:09 15:27:29 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" 2026/11/01 15:32:09 15:27:29 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" 2026/11/01 15:32:09 15:27:29 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: Created - Container created 2026/11/01 15:32:09 15:27:29 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" in 868ms (868ms including waiting). Image size: 187802205 bytes. 2026/11/01 15:32:09 15:27:28 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: Started - Container started 2026/11/01 15:32:09 15:27:28 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" 2026/11/01 15:32:09 15:27:28 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: Created - Container created 2026/11/01 15:32:09 15:27:28 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" in 967ms (967ms including waiting). Image size: 241242949 bytes. 2026/11/01 15:32:09 15:27:28 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" in 978ms (978ms including waiting). Image size: 241242949 bytes. 2026/11/01 15:32:09 15:27:28 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: Created - Container created 2026/11/01 15:32:09 15:27:28 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" 2026/11/01 15:32:09 15:27:28 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: Started - Container started 2026/11/01 15:32:09 15:27:27 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" in 961ms (961ms including waiting). Image size: 182438697 bytes. 2026/11/01 15:32:09 15:27:27 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: Created - Container created 2026/11/01 15:32:09 15:27:27 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" in 919ms (919ms including waiting). Image size: 182438697 bytes. 2026/11/01 15:32:09 15:27:27 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: Created - Container created 2026/11/01 15:32:09 15:27:27 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: Started - Container started 2026/11/01 15:32:09 15:27:27 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" 2026/11/01 15:32:09 15:27:27 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: Started - Container started 2026/11/01 15:32:09 15:27:27 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" 2026/11/01 15:32:09 15:27:26 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: Pulled - Container image "quay.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:1e614e46799fd020ac18d95c06a463421154ce86e59ccb3fa6007e66fed9910e" already present on machine and can be accessed by the pod 2026/11/01 15:32:09 15:27:26 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: Pulled - Successfully pulled image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" in 841ms (841ms including waiting). Image size: 799822033 bytes. 2026/11/01 15:32:09 15:27:26 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: Created - Container created 2026/11/01 15:32:09 15:27:26 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: Started - Container started 2026/11/01 15:32:09 15:27:26 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" 2026/11/01 15:32:09 15:27:26 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" 2026/11/01 15:32:09 15:27:26 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: Started - Container started 2026/11/01 15:32:09 15:27:26 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: Created - Container created 2026/11/01 15:32:09 15:27:26 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: Pulled - Container image "quay.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:1e614e46799fd020ac18d95c06a463421154ce86e59ccb3fa6007e66fed9910e" already present on machine and can be accessed by the pod 2026/11/01 15:32:09 15:27:26 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: Started - Container started 2026/11/01 15:32:09 15:27:26 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: Created - Container created 2026/11/01 15:32:09 15:27:26 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: Pulled - Successfully pulled image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" in 1.006s (1.006s including waiting). Image size: 799822033 bytes. 2026/11/01 15:32:09 15:27:26 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: Created - Container created 2026/11/01 15:32:09 15:27:26 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: Started - Container started 2026/11/01 15:32:09 15:27:25 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-674b4b9c7-nhhvl to ip-10-0-1-85.us-west-2.compute.internal 2026/11/01 15:32:09 15:27:25 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: AddedInterface - Add eth0 [10.128.1.252/23] from ovn-kubernetes 2026/11/01 15:32:09 15:27:25 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: Pulling - Pulling image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" 2026/11/01 15:32:09 15:27:25 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: Pulling - Pulling image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" 2026/11/01 15:32:09 15:27:25 INFO Pod/rhods-dashboard-674b4b9c7-nhhvl: AddedInterface - Add eth0 [10.129.0.163/23] from ovn-kubernetes 2026/11/01 15:32:09 15:27:24 INFO Pod/rhods-dashboard-674b4b9c7-2rrdk: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-674b4b9c7-2rrdk to ip-10-0-1-150.us-west-2.compute.internal 2026/11/01 15:32:09 15:27:24 INFO ReplicaSet/rhods-dashboard-674b4b9c7: SuccessfulCreate - Created pod: rhods-dashboard-674b4b9c7-2rrdk 2026/11/01 15:32:09 15:27:24 INFO ReplicaSet/rhods-dashboard-674b4b9c7: SuccessfulCreate - Created pod: rhods-dashboard-674b4b9c7-nhhvl 2026/11/01 15:32:09 15:27:24 INFO Deployment/rhods-dashboard: ScalingReplicaSet - Scaled up replica set rhods-dashboard-674b4b9c7 from 0 to 2 2026/11/01 15:32:09 15:27:08 INFO Pod/maas-controller-86f7886655-q7rhp: Killing - Stopping container manager 2026/11/01 15:32:09 15:27:08 INFO Pod/maas-api-759b8d4d45-xkppk: Killing - Stopping container maas-api 2026/11/01 15:32:09 === RESOURCE QUOTAS === 2026/11/01 15:32:09 No resource quotas found in monitored namespaces 2026/11/01 15:32:09 Classification for group 2: infrastructure/storage (code=1006, confidence=medium) 2026/11/01 15:32:09 Evidence: event Pod/workload-variant-autoscaler-controller-manager-54bc958497-mwhxx: FailedMount - MountVolume.SetUp failed for volume "epp-metrics-token" : secret "workload-variant-autoscaler-epp-metrics-token" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/workload-variant-autoscaler-controller-manager-54bc958497-mwhxx: FailedMount - MountVolume.SetUp failed for volume \"epp-metrics-token\" : secret \"workload-variant-autoscaler-epp-metrics-token\" not found"],"confidence":"medium"} 2026/11/01 15:32:09 Diagnostics complete for failed test: group 2 controller_test.go:753: Stopping: group 2 test failed. 2026/11/01 15:32:09 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/11/01 15:32:09 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/11/01 15:36:26 Circuit breaker: success recorded, resetting failure counter (was 1) 2026/11/01 15:45:22 TEST FAILURE DETECTED: group 4 2026/11/01 15:45:22 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 4) === 2026/11/01 15:45:23 === CLUSTER STATE === 2026/11/01 15:45:23 Node: ip-10-0-1-134.us-west-2.compute.internal 2026/11/01 15:45:23 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/11/01 15:45:23 Condition DiskPressure: False - kubelet has no disk pressure 2026/11/01 15:45:23 Condition PIDPressure: False - kubelet has sufficient PID available 2026/11/01 15:45:23 Condition Ready: True - kubelet is posting ready status 2026/11/01 15:45:23 Resources: 15500m CPU, 63870868Ki memory allocatable (of 16000m CPU, 65021844Ki memory capacity) 2026/11/01 15:45:23 Node: ip-10-0-1-150.us-west-2.compute.internal 2026/11/01 15:45:23 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/11/01 15:45:23 Condition DiskPressure: False - kubelet has no disk pressure 2026/11/01 15:45:23 Condition PIDPressure: False - kubelet has sufficient PID available 2026/11/01 15:45:23 Condition Ready: True - kubelet is posting ready status 2026/11/01 15:45:23 Resources: 15500m CPU, 63870872Ki memory allocatable (of 16000m CPU, 65021848Ki memory capacity) 2026/11/01 15:45:23 Node: ip-10-0-1-85.us-west-2.compute.internal 2026/11/01 15:45:23 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/11/01 15:45:23 Condition DiskPressure: False - kubelet has no disk pressure 2026/11/01 15:45:23 Condition PIDPressure: False - kubelet has sufficient PID available 2026/11/01 15:45:23 Condition Ready: True - kubelet is posting ready status 2026/11/01 15:45:23 Resources: 15500m CPU, 63870868Ki memory allocatable (of 16000m CPU, 65021844Ki memory capacity) 2026/11/01 15:45:23 === DEPLOYMENTS === 2026/11/01 15:45:23 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/11/01 15:45:23 === PODS === 2026/11/01 15:45:23 Failed to collect pod data: redhat-ods-applications/maas-controller-86f7886655-nmzs4: container manager not ready 2026/11/01 15:45:23 === OPERATOR STATUS === 2026/11/01 15:45:23 Operator deployment: 3/3 ready 2026/11/01 15:45:23 === DSCI STATUS === 2026/11/01 15:45:23 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/11/01 15:45:23 === DSC STATUS === 2026/11/01 15:45:23 DSC default-dsc: 2026/11/01 15:45:23 Ready: True - 2026/11/01 15:45:23 ProvisioningSucceeded: True - 2026/11/01 15:45:23 ComponentsReady: True - 2026/11/01 15:45:23 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/11/01 15:45:23 DashboardReady: False - Component ManagementState is set to Removed 2026/11/01 15:45:23 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/11/01 15:45:23 KserveLLMInferenceServiceDependencies: True - 2026/11/01 15:45:23 KserveLLMInferenceServiceWideEPDependencies: True - 2026/11/01 15:45:23 KserveReady: True - 2026/11/01 15:45:23 KueueReady: True - 2026/11/01 15:45:23 LLM-D-WVADependencies: True - 2026/11/01 15:45:23 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/11/01 15:45:23 ModelControllerReady: True - 2026/11/01 15:45:23 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/11/01 15:45:23 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/11/01 15:45:23 OGXReady: False - Component ManagementState is set to Removed 2026/11/01 15:45:23 RayReady: False - Component ManagementState is set to Removed 2026/11/01 15:45:23 SparkOperatorReady: True - 2026/11/01 15:45:23 TrainerReady: False - Component ManagementState is set to Removed 2026/11/01 15:45:23 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/11/01 15:45:23 TrustyAIReady: False - Component ManagementState is set to Removed 2026/11/01 15:45:23 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/11/01 15:45:23 === RECENT EVENTS === 2026/11/01 15:45:23 15:45:04 INFO Job/maas-api-key-cleanup-29892465: Completed - Job completed 2026/11/01 15:45:23 15:45:04 INFO CronJob/maas-api-key-cleanup: SawCompletedJob - Saw completed job: maas-api-key-cleanup-29892465, condition: Complete 2026/11/01 15:45:23 15:45:01 INFO Pod/maas-api-key-cleanup-29892465-f7r7h: Created - Container created 2026/11/01 15:45:23 15:45:01 INFO Pod/maas-api-key-cleanup-29892465-f7r7h: Started - Container started 2026/11/01 15:45:23 15:45:00 INFO Pod/maas-api-key-cleanup-29892465-f7r7h: Scheduled - Successfully assigned redhat-ods-applications/maas-api-key-cleanup-29892465-f7r7h to ip-10-0-1-150.us-west-2.compute.internal 2026/11/01 15:45:23 15:45:00 INFO Job/maas-api-key-cleanup-29892465: SuccessfulCreate - Created pod: maas-api-key-cleanup-29892465-f7r7h 2026/11/01 15:45:23 15:45:00 INFO Pod/maas-api-key-cleanup-29892465-f7r7h: Pulled - Container image "registry.redhat.io/ubi9/ubi-minimal@sha256:907b68736aa798b2d38255b7aa070b2a70acb90803864a40f05d0ec47556ddd0" already present on machine and can be accessed by the pod 2026/11/01 15:45:23 15:45:00 INFO CronJob/maas-api-key-cleanup: SuccessfulCreate - Created job maas-api-key-cleanup-29892465 2026/11/01 15:45:23 15:45:00 INFO Pod/maas-api-key-cleanup-29892465-f7r7h: AddedInterface - Add eth0 [10.128.0.60/23] from ovn-kubernetes 2026/11/01 15:45:23 15:44:20 INFO Pod/maas-controller-86f7886655-nmzs4: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 964ms (964ms including waiting). Image size: 212866077 bytes. 2026/11/01 15:45:23 15:44:19 INFO Pod/maas-controller-86f7886655-nmzs4: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/11/01 15:45:23 15:44:19 WARN Pod/maas-controller-86f7886655-nmzs4: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-rps8z); attempting to pull the image may not succeed. 2026/11/01 15:45:23 15:41:12 INFO Pod/maas-controller-86f7886655-nmzs4: Created - Container created 2026/11/01 15:45:23 15:41:12 INFO Pod/maas-controller-86f7886655-nmzs4: Started - Container started 2026/11/01 15:45:23 15:41:12 INFO Pod/maas-controller-86f7886655-nmzs4: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 1.001s (1.001s including waiting). Image size: 212866077 bytes. 2026/11/01 15:45:23 === RESOURCE QUOTAS === 2026/11/01 15:45:23 No resource quotas found in monitored namespaces 2026/11/01 15:45:23 Classification for group 4: infrastructure/pod-startup (code=1002, confidence=medium) 2026/11/01 15:45:23 Evidence: container maas-controller-86f7886655-nmzs4/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-86f7886655-nmzs4_redhat-ods-applications(b63d7f23-6f6f-429d-b0eb-7e8913a6491c) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-86f7886655-nmzs4/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-86f7886655-nmzs4_redhat-ods-applications(b63d7f23-6f6f-429d-b0eb-7e8913a6491c)"],"confidence":"medium"} 2026/11/01 15:45:23 Diagnostics complete for failed test: group 4 controller_test.go:753: Stopping: group 4 test failed. 2026/11/01 15:45:23 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/11/01 15:45:23 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/11/01 15:46:36 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (1478.29s) === RUN TestOdhOperator 2026/11/01 15:22:58 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/11/01 15:47:39 TEST FAILURE DETECTED: components 2026/11/01 15:47:39 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/11/01 15:47:40 === CLUSTER STATE === 2026/11/01 15:47:40 Node: ip-10-0-1-134.us-west-2.compute.internal 2026/11/01 15:47:40 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/11/01 15:47:40 Condition DiskPressure: False - kubelet has no disk pressure 2026/11/01 15:47:40 Condition PIDPressure: False - kubelet has sufficient PID available 2026/11/01 15:47:40 Condition Ready: True - kubelet is posting ready status 2026/11/01 15:47:40 Resources: 15500m CPU, 63870868Ki memory allocatable (of 16000m CPU, 65021844Ki memory capacity) 2026/11/01 15:47:40 Node: ip-10-0-1-150.us-west-2.compute.internal 2026/11/01 15:47:40 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/11/01 15:47:40 Condition DiskPressure: False - kubelet has no disk pressure 2026/11/01 15:47:40 Condition PIDPressure: False - kubelet has sufficient PID available 2026/11/01 15:47:40 Condition Ready: True - kubelet is posting ready status 2026/11/01 15:47:40 Resources: 15500m CPU, 63870872Ki memory allocatable (of 16000m CPU, 65021848Ki memory capacity) 2026/11/01 15:47:40 Node: ip-10-0-1-85.us-west-2.compute.internal 2026/11/01 15:47:40 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/11/01 15:47:40 Condition DiskPressure: False - kubelet has no disk pressure 2026/11/01 15:47:40 Condition PIDPressure: False - kubelet has sufficient PID available 2026/11/01 15:47:40 Condition Ready: True - kubelet is posting ready status 2026/11/01 15:47:40 Resources: 15500m CPU, 63870868Ki memory allocatable (of 16000m CPU, 65021844Ki memory capacity) 2026/11/01 15:47:40 === DEPLOYMENTS === 2026/11/01 15:47:40 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/11/01 15:47:40 === PODS === 2026/11/01 15:47:40 Failed to collect pod data: redhat-ods-applications/maas-controller-86f7886655-nmzs4: container manager not ready 2026/11/01 15:47:40 === OPERATOR STATUS === 2026/11/01 15:47:40 Operator deployment: 3/3 ready 2026/11/01 15:47:40 === DSCI STATUS === 2026/11/01 15:47:40 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/11/01 15:47:40 === DSC STATUS === 2026/11/01 15:47:40 DSC default-dsc: 2026/11/01 15:47:40 Ready: True - 2026/11/01 15:47:40 ProvisioningSucceeded: True - 2026/11/01 15:47:40 ComponentsReady: True - 2026/11/01 15:47:40 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/11/01 15:47:40 DashboardReady: False - Component ManagementState is set to Removed 2026/11/01 15:47:40 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/11/01 15:47:40 KserveLLMInferenceServiceDependencies: True - 2026/11/01 15:47:40 KserveLLMInferenceServiceWideEPDependencies: True - 2026/11/01 15:47:40 KserveReady: True - 2026/11/01 15:47:40 KueueReady: True - 2026/11/01 15:47:40 LLM-D-WVADependencies: True - 2026/11/01 15:47:40 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/11/01 15:47:40 ModelControllerReady: True - 2026/11/01 15:47:40 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/11/01 15:47:40 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/11/01 15:47:40 OGXReady: False - Component ManagementState is set to Removed 2026/11/01 15:47:40 RayReady: False - Component ManagementState is set to Removed 2026/11/01 15:47:40 SparkOperatorReady: True - 2026/11/01 15:47:40 TrainerReady: True - 2026/11/01 15:47:40 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/11/01 15:47:40 TrustyAIReady: False - Component ManagementState is set to Removed 2026/11/01 15:47:40 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/11/01 15:47:40 === RECENT EVENTS === 2026/11/01 15:47:40 15:47:23 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-78d4c86675-xxhl7_60c1a737-1772-4347-9afa-b68c36db9e00 became leader 2026/11/01 15:47:40 15:47:21 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-8558c8ff56-kgz6j_ca645118-4070-40d9-9355-9fc5642b8eee became leader 2026/11/01 15:47:40 15:47:08 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-c6d898f49-gt564_af06ad3d-0c5f-40c4-9eab-d3f2218f6398 became leader 2026/11/01 15:47:40 15:47:04 INFO Pod/llmisvc-controller-manager-78d4c86675-xxhl7: Started - Container started 2026/11/01 15:47:40 15:47:04 INFO Pod/kserve-controller-manager-8558c8ff56-kgz6j: Created - Container created 2026/11/01 15:47:40 15:47:04 INFO Pod/kserve-controller-manager-8558c8ff56-kgz6j: Started - Container started 2026/11/01 15:47:40 15:47:04 INFO Pod/kserve-controller-manager-8558c8ff56-kgz6j: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 1.026s (1.027s including waiting). Image size: 238435613 bytes. 2026/11/01 15:47:40 15:47:04 INFO Pod/llmisvc-controller-manager-78d4c86675-xxhl7: Created - Container created 2026/11/01 15:47:40 15:47:04 INFO Pod/llmisvc-controller-manager-78d4c86675-xxhl7: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 926ms (926ms including waiting). Image size: 236366599 bytes. 2026/11/01 15:47:40 15:47:03 INFO Pod/llmisvc-controller-manager-78d4c86675-xxhl7: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/11/01 15:47:40 15:47:03 INFO Pod/llmisvc-controller-manager-78d4c86675-xxhl7: AddedInterface - Add eth0 [10.128.0.68/23] from ovn-kubernetes 2026/11/01 15:47:40 15:47:03 INFO Pod/kserve-controller-manager-8558c8ff56-kgz6j: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/11/01 15:47:40 15:47:03 INFO Pod/kserve-controller-manager-8558c8ff56-kgz6j: AddedInterface - Add eth0 [10.128.0.67/23] from ovn-kubernetes 2026/11/01 15:47:40 15:47:02 INFO Pod/llmisvc-controller-manager-78d4c86675-xxhl7: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-78d4c86675-xxhl7 to ip-10-0-1-150.us-west-2.compute.internal 2026/11/01 15:47:40 15:47:02 INFO Pod/kserve-controller-manager-8558c8ff56-kgz6j: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-8558c8ff56-kgz6j to ip-10-0-1-150.us-west-2.compute.internal 2026/11/01 15:47:40 15:47:02 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-8558c8ff56 from 0 to 1 2026/11/01 15:47:40 15:47:02 INFO ReplicaSet/llmisvc-controller-manager-78d4c86675: SuccessfulCreate - Created pod: llmisvc-controller-manager-78d4c86675-xxhl7 2026/11/01 15:47:40 15:47:02 INFO ReplicaSet/kserve-controller-manager-8558c8ff56: SuccessfulCreate - Created pod: kserve-controller-manager-8558c8ff56-kgz6j 2026/11/01 15:47:40 15:47:02 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-78d4c86675 from 0 to 1 2026/11/01 15:47:40 15:46:51 INFO Pod/model-serving-api-5f9bc6947d-9zbpk: Started - Container started 2026/11/01 15:47:40 15:46:51 INFO Pod/model-serving-api-5f9bc6947d-9zbpk: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine and can be accessed by the pod 2026/11/01 15:47:40 15:46:51 INFO Pod/odh-model-controller-c6d898f49-gt564: Created - Container created 2026/11/01 15:47:40 15:46:51 INFO Pod/odh-model-controller-c6d898f49-gt564: Started - Container started 2026/11/01 15:47:40 15:46:51 INFO Pod/model-serving-api-5f9bc6947d-9zbpk: AddedInterface - Add eth0 [10.128.0.66/23] from ovn-kubernetes 2026/11/01 15:47:40 15:46:51 INFO Pod/model-serving-api-5f9bc6947d-9zbpk: Created - Container created 2026/11/01 15:47:40 15:46:50 INFO Pod/model-serving-api-5f9bc6947d-9zbpk: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5f9bc6947d-9zbpk to ip-10-0-1-150.us-west-2.compute.internal 2026/11/01 15:47:40 15:46:50 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5f9bc6947d from 0 to 1 2026/11/01 15:47:40 15:46:50 INFO ReplicaSet/model-serving-api-5f9bc6947d: SuccessfulCreate - Created pod: model-serving-api-5f9bc6947d-9zbpk 2026/11/01 15:47:40 15:46:50 INFO Pod/odh-model-controller-c6d898f49-gt564: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/11/01 15:47:40 15:46:50 WARN ReplicaSet/model-serving-api-5f9bc6947d: FailedCreate - Error creating: pods "model-serving-api-5f9bc6947d-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/11/01 15:47:40 15:46:50 INFO Pod/odh-model-controller-c6d898f49-gt564: AddedInterface - Add eth0 [10.128.0.65/23] from ovn-kubernetes 2026/11/01 15:47:40 15:46:50 INFO Pod/odh-model-controller-c6d898f49-gt564: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 883ms (883ms including waiting). Image size: 238959091 bytes. 2026/11/01 15:47:40 15:46:49 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/11/01 15:47:40 15:46:49 WARN Pod/odh-model-controller-c6d898f49-gt564: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/11/01 15:47:40 15:46:48 INFO Pod/odh-model-controller-c6d898f49-gt564: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-c6d898f49-gt564 to ip-10-0-1-150.us-west-2.compute.internal 2026/11/01 15:47:40 15:46:48 INFO ReplicaSet/odh-model-controller-c6d898f49: SuccessfulCreate - Created pod: odh-model-controller-c6d898f49-gt564 2026/11/01 15:47:40 15:46:48 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-c6d898f49 from 0 to 1 2026/11/01 15:47:40 15:46:40 INFO Pod/odh-model-controller-c6d898f49-cswqz: Killing - Stopping container manager 2026/11/01 15:47:40 15:46:40 INFO Pod/llmisvc-controller-manager-78d4c86675-nfgm5: Killing - Stopping container manager 2026/11/01 15:47:40 15:46:40 INFO Pod/model-serving-api-5f9bc6947d-mt9qx: Killing - Stopping container server 2026/11/01 15:47:40 15:46:39 INFO Pod/kserve-controller-manager-8558c8ff56-s9r56: Killing - Stopping container manager 2026/11/01 15:47:40 15:46:39 WARN Endpoints/kserve-webhook-server-service: FailedToUpdateEndpoint - Failed to update endpoint redhat-ods-applications/kserve-webhook-server-service: Operation cannot be fulfilled on endpoints "kserve-webhook-server-service": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/kserve-webhook-server-service, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 5ca40938-ffd4-476d-be8c-122db2ec7861, UID in object meta: 2026/11/01 15:47:40 15:46:17 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-8558c8ff56-s9r56_61ffc68d-e1cf-43aa-8bc1-77e47b400822 became leader 2026/11/01 15:47:40 15:46:17 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-78d4c86675-nfgm5_5defb2d2-230f-48d2-9dac-599c74b8a86e became leader 2026/11/01 15:47:40 15:46:01 INFO Pod/llmisvc-controller-manager-78d4c86675-nfgm5: Created - Container created 2026/11/01 15:47:40 15:46:01 INFO Pod/kserve-controller-manager-8558c8ff56-s9r56: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 1.124s (1.124s including waiting). Image size: 238435613 bytes. 2026/11/01 15:47:40 15:46:01 INFO Pod/kserve-controller-manager-8558c8ff56-s9r56: Started - Container started 2026/11/01 15:47:40 15:46:01 INFO Pod/llmisvc-controller-manager-78d4c86675-nfgm5: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 1.03s (1.03s including waiting). Image size: 236366599 bytes. 2026/11/01 15:47:40 15:46:01 INFO Pod/llmisvc-controller-manager-78d4c86675-nfgm5: Started - Container started 2026/11/01 15:47:40 15:46:01 INFO Pod/kserve-controller-manager-8558c8ff56-s9r56: Created - Container created 2026/11/01 15:47:40 15:45:59 INFO Pod/llmisvc-controller-manager-78d4c86675-nfgm5: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-78d4c86675-nfgm5 to ip-10-0-1-150.us-west-2.compute.internal 2026/11/01 15:47:40 15:45:59 INFO Pod/kserve-controller-manager-8558c8ff56-s9r56: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-8558c8ff56-s9r56 to ip-10-0-1-150.us-west-2.compute.internal 2026/11/01 15:47:40 15:45:59 INFO Pod/llmisvc-controller-manager-78d4c86675-nfgm5: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/11/01 15:47:40 15:45:59 INFO ReplicaSet/llmisvc-controller-manager-78d4c86675: SuccessfulCreate - Created pod: llmisvc-controller-manager-78d4c86675-nfgm5 2026/11/01 15:47:40 15:45:59 INFO Pod/kserve-controller-manager-8558c8ff56-s9r56: AddedInterface - Add eth0 [10.128.0.63/23] from ovn-kubernetes 2026/11/01 15:47:40 15:45:59 INFO ReplicaSet/kserve-controller-manager-8558c8ff56: SuccessfulCreate - Created pod: kserve-controller-manager-8558c8ff56-s9r56 2026/11/01 15:47:40 15:45:59 INFO Pod/kserve-controller-manager-8558c8ff56-s9r56: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/11/01 15:47:40 15:45:59 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-78d4c86675 from 0 to 1 2026/11/01 15:47:40 15:45:59 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-8558c8ff56 from 0 to 1 2026/11/01 15:47:40 15:45:59 INFO Pod/llmisvc-controller-manager-78d4c86675-nfgm5: AddedInterface - Add eth0 [10.128.0.64/23] from ovn-kubernetes 2026/11/01 15:47:40 15:45:54 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-c6d898f49-cswqz_ee94e1f6-b8d5-4ac5-9cbb-c728ac19af55 became leader 2026/11/01 15:47:40 15:45:49 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-55cf4d84cd-rgk95_7a218662-50f6-4fd0-887c-599880df0092 became leader 2026/11/01 15:47:40 15:45:38 INFO Pod/model-serving-api-5f9bc6947d-mt9qx: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5f9bc6947d-mt9qx to ip-10-0-1-150.us-west-2.compute.internal 2026/11/01 15:47:40 15:45:38 WARN ReplicaSet/model-serving-api-5f9bc6947d: FailedCreate - Error creating: pods "model-serving-api-5f9bc6947d-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/11/01 15:47:40 15:45:38 INFO Pod/odh-model-controller-c6d898f49-cswqz: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 989ms (989ms including waiting). Image size: 238959091 bytes. 2026/11/01 15:47:40 15:45:38 INFO Pod/odh-model-controller-c6d898f49-cswqz: Created - Container created 2026/11/01 15:47:40 15:45:38 INFO Pod/odh-model-controller-c6d898f49-cswqz: Started - Container started 2026/11/01 15:47:40 15:45:38 INFO Pod/model-serving-api-5f9bc6947d-mt9qx: AddedInterface - Add eth0 [10.128.0.62/23] from ovn-kubernetes 2026/11/01 15:47:40 15:45:38 INFO Pod/model-serving-api-5f9bc6947d-mt9qx: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine and can be accessed by the pod 2026/11/01 15:47:40 15:45:38 INFO Pod/model-serving-api-5f9bc6947d-mt9qx: Created - Container created 2026/11/01 15:47:40 15:45:38 INFO Pod/model-serving-api-5f9bc6947d-mt9qx: Started - Container started 2026/11/01 15:47:40 15:45:38 INFO ReplicaSet/model-serving-api-5f9bc6947d: SuccessfulCreate - Created pod: model-serving-api-5f9bc6947d-mt9qx 2026/11/01 15:47:40 15:45:37 INFO Pod/odh-model-controller-c6d898f49-cswqz: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/11/01 15:47:40 15:45:37 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5f9bc6947d from 0 to 1 2026/11/01 15:47:40 15:45:37 INFO Pod/odh-model-controller-c6d898f49-cswqz: AddedInterface - Add eth0 [10.128.0.61/23] from ovn-kubernetes 2026/11/01 15:47:40 15:45:36 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/11/01 15:47:40 15:45:36 INFO Pod/odh-model-controller-c6d898f49-cswqz: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-c6d898f49-cswqz to ip-10-0-1-150.us-west-2.compute.internal 2026/11/01 15:47:40 15:45:36 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-c6d898f49 from 0 to 1 2026/11/01 15:47:40 15:45:36 WARN Pod/odh-model-controller-c6d898f49-cswqz: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/11/01 15:47:40 15:45:36 INFO ReplicaSet/odh-model-controller-c6d898f49: SuccessfulCreate - Created pod: odh-model-controller-c6d898f49-cswqz 2026/11/01 15:47:40 15:45:30 INFO Pod/kubeflow-trainer-controller-manager-55cf4d84cd-rgk95: Started - Container started 2026/11/01 15:47:40 15:45:30 INFO Pod/kubeflow-trainer-controller-manager-55cf4d84cd-rgk95: AddedInterface - Add eth0 [10.130.0.138/23] from ovn-kubernetes 2026/11/01 15:47:40 15:45:30 INFO Pod/kubeflow-trainer-controller-manager-55cf4d84cd-rgk95: Pulled - Container image "quay.io/rhoai/odh-trainer-rhel9@sha256:59ea523e2df9c1e705c755308ac11420682081cc63d54890e2b15fbd7d8ba570" already present on machine and can be accessed by the pod 2026/11/01 15:47:40 15:45:30 INFO Pod/kubeflow-trainer-controller-manager-55cf4d84cd-rgk95: Created - Container created 2026/11/01 15:47:40 15:45:29 INFO Pod/kubeflow-trainer-controller-manager-55cf4d84cd-rgk95: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-55cf4d84cd-rgk95 to ip-10-0-1-134.us-west-2.compute.internal 2026/11/01 15:47:40 15:45:29 INFO ReplicaSet/kubeflow-trainer-controller-manager-55cf4d84cd: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-55cf4d84cd-rgk95 2026/11/01 15:47:40 15:45:29 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-55cf4d84cd from 0 to 1 2026/11/01 15:47:40 15:45:26 INFO Pod/llmisvc-controller-manager-78d4c86675-fcxwm: Killing - Stopping container manager 2026/11/01 15:47:40 15:45:26 INFO Pod/model-serving-api-5f9bc6947d-d46pb: Killing - Stopping container server 2026/11/01 15:47:40 15:45:26 INFO Pod/kserve-controller-manager-8558c8ff56-pzqhr: Killing - Stopping container manager 2026/11/01 15:47:40 15:45:26 INFO Pod/odh-model-controller-c6d898f49-28mq8: Killing - Stopping container manager 2026/11/01 15:47:40 15:45:04 INFO CronJob/maas-api-key-cleanup: SawCompletedJob - Saw completed job: maas-api-key-cleanup-29892465, condition: Complete 2026/11/01 15:47:40 15:45:04 INFO Job/maas-api-key-cleanup-29892465: Completed - Job completed 2026/11/01 15:47:40 15:45:01 INFO Pod/maas-api-key-cleanup-29892465-f7r7h: Created - Container created 2026/11/01 15:47:40 15:45:01 INFO Pod/maas-api-key-cleanup-29892465-f7r7h: Started - Container started 2026/11/01 15:47:40 15:45:00 INFO Pod/maas-api-key-cleanup-29892465-f7r7h: Scheduled - Successfully assigned redhat-ods-applications/maas-api-key-cleanup-29892465-f7r7h to ip-10-0-1-150.us-west-2.compute.internal 2026/11/01 15:47:40 15:45:00 INFO Pod/maas-api-key-cleanup-29892465-f7r7h: Pulled - Container image "registry.redhat.io/ubi9/ubi-minimal@sha256:907b68736aa798b2d38255b7aa070b2a70acb90803864a40f05d0ec47556ddd0" already present on machine and can be accessed by the pod 2026/11/01 15:47:40 15:45:00 INFO Pod/maas-api-key-cleanup-29892465-f7r7h: AddedInterface - Add eth0 [10.128.0.60/23] from ovn-kubernetes 2026/11/01 15:47:40 15:45:00 INFO Job/maas-api-key-cleanup-29892465: SuccessfulCreate - Created pod: maas-api-key-cleanup-29892465-f7r7h 2026/11/01 15:47:40 15:45:00 INFO CronJob/maas-api-key-cleanup: SuccessfulCreate - Created job maas-api-key-cleanup-29892465 2026/11/01 15:47:40 15:44:20 INFO Pod/maas-controller-86f7886655-nmzs4: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 964ms (964ms including waiting). Image size: 212866077 bytes. 2026/11/01 15:47:40 15:44:19 INFO Pod/maas-controller-86f7886655-nmzs4: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/11/01 15:47:40 15:44:19 WARN Pod/maas-controller-86f7886655-nmzs4: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-rps8z); attempting to pull the image may not succeed. 2026/11/01 15:47:40 === RESOURCE QUOTAS === 2026/11/01 15:47:40 No resource quotas found in monitored namespaces 2026/11/01 15:47:40 Classification for components: infrastructure/pod-startup (code=1002, confidence=medium) 2026/11/01 15:47:40 Evidence: container maas-controller-86f7886655-nmzs4/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-86f7886655-nmzs4_redhat-ods-applications(b63d7f23-6f6f-429d-b0eb-7e8913a6491c) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-86f7886655-nmzs4/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-86f7886655-nmzs4_redhat-ods-applications(b63d7f23-6f6f-429d-b0eb-7e8913a6491c)"],"confidence":"medium"} 2026/11/01 15:47:40 Diagnostics complete for failed test: components controller_test.go:484: Stopping: components test failed. 2026/11/01 15:47:40 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/11/01 15:47:40 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/11/01 15:47:43 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (1486.06s)