=== RUN TestOdhOperator/components/group_2/kueue/Validate_component_removed_to_unmanaged_transition helper_test.go:150: Setting Kueue component (default-kueue) to Removed mode to start with a clean state. 2026-07-16T13:39:39Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.20.12"}} helper_test.go:150: Cleaning up any existing Kueue test resources. helper_test.go:150: Cleaning up Kueue cluster-scoped resources helper_test.go:150: Will attempt to delete 2 Kueue resources helper_test.go:150: Attempting to delete ClusterQueue /default helper_test.go:150: Successfully processed deletion of ClusterQueue /default helper_test.go:150: Attempting to delete Kueue /cluster helper_test.go:150: Successfully processed deletion of Kueue /cluster helper_test.go:150: Creating test namespace (test-kueue-managed-d9cdu6ont5ns7b6lic0g) with Kueue management annotation. helper_test.go:150: Creating Kueue ConfigMap (kueue-manager-config) to verify default Kueue resource is created correctly. helper_test.go:150: Creating Kueue ConfigMap (kueue-manager-config) in namespace (redhat-ods-applications). helper_test.go:150: Updating the management state of the component in the DataScienceCluster to Unmanaged (expecting NotReady initially due to missing operator). helper_test.go:150: Ensuring LWS Operator is installed (namespace=openshift-lws-operator, name=leader-worker-set). helper_test.go:150: Ensuring LeaderWorkerSetOperator CR exists (namespace=openshift-lws-operator, name=cluster). helper_test.go:150: Ensuring OCP Kueue Operator is installed (namespace=openshift-kueue-operator, name=kueue-operator). helper_test.go:150: Verifying the component in the DataScienceCluster transitions to Ready state. helper_test.go:150: Verifying Kueue configuration (cluster) is created with all expected frameworks. testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_2/kueue/Validate_component_removed_to_unmanaged_transition (30.86s) === RUN TestOdhOperator/components/group_2/kueue === PAUSE TestOdhOperator/components/group_2/kueue === CONT TestOdhOperator/components/group_2/kueue test_context_test.go:223: Timed out after 10.000s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:114 with: Expected resource 'cluster' of kind 'Kueue' to match condition {(["BatchJob", "Deployment", "JobSet", "LeaderWorkerSet", "MPIJob", "PaddleJob", "Pod", "PyTorchJob", "RayCluster", "RayJob", "StatefulSet", "TFJob", "TrainJob", "XGBoostJob"] - .spec.config.integrations.frameworks) | length == 0 []}. Expected <string>: map[apiVersion:kueue.openshift.io/v1 kind:Kueue metadata:map[annotations:map[platform.opendatahub.io/instance.generation:1 platform.opendatahub.io/instance.name:default-kueue platform.opendatahub.io/instance.uid:cdef0d72-c324-42ff-9ab5-fa07b6861b1d platform.opendatahub.io/type:OpenShift AI Self-Managed platform.opendatahub.io/version:3.5.0-ea.2] creationTimestamp:2026-07-16T12:48:36Z deletionGracePeriodSeconds:0 deletionTimestamp:2026-07-16T13:39:50Z finalizers:[kueue.openshift.io/finalizer] generation:2 labels:map[platform.opendatahub.io/part-of:kueue] managedFields:[map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:status:map[f:conditions:map[k:{"type":"Available"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"CertManagerAvailable"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"Degraded"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"Progressing"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]]] f:readyReplicas:map[]]] manager:kueue-operator operation:Apply subresource:status time:2026-07-16T12:49:20Z] map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[.:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:labels:map[.:map[] f:platform.opendatahub.io/part-of:map[]]] f:spec:map[.:map[] f:config:map[.:map[] f:integrations:map[.:map[] f:frameworks:map[.:map[] v:"Deployment":map[] v:"Pod":map[] v:"PyTorchJob":map[] v:"RayCluster":map[] v:"RayJob":map[] v:"StatefulSet":map[] v:"TrainJob":map[]]]] f:logLevel:map[] f:managementState:map[] f:operatorLogLevel:map[]]] manager:manager operation:Update time:2026-07-16T12:48:36Z] map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:finalizers:map[.:map[] v:"kueue.openshift.io/finalizer":map[]]]] manager:kueue-operator operation:Update time:2026-07-16T13:39:50Z]] name:cluster resourceVersion:221292518 uid:3a71e763-9b40-44d3-8cd5-adc7065823e5] spec:map[config:map[integrations:map[frameworks:[Deployment Pod PyTorchJob RayCluster RayJob StatefulSet TrainJob]]] logLevel:Normal managementState:Managed operatorLogLevel:Normal] status:map[conditions:[map[lastTransitionTime:2026-07-16T12:49:20Z message:All 2 replicas are ready reason:AllReplicasReady status:True type:Available] map[lastTransitionTime:2026-07-16T12:49:20Z message:Deployment is up to date reason:AsExpected status:False type:Progressing] map[lastTransitionTime:2026-07-16T12:49:20Z message: reason:AsExpected status:False type:Degraded] map[lastTransitionTime:2026-07-16T12:49:03Z message:cert-manager is installed reason:CertManagerInstalled status:True type:CertManagerAvailable]] readyReplicas:2]] to match expression <string>: (["BatchJob", "Deployment", "JobSet", "LeaderWorkerSet", "MPIJob", "PaddleJob", "Pod", "PyTorchJob", "RayCluster", "RayJob", "StatefulSet", "TFJob", "TrainJob", "XGBoostJob"] - .spec.config.integrations.frameworks) | length == 0 test_context_test.go:1106: cleanup: best-effort delete failed for /, Kind= redhat-ods-applications/kueue-manager-config: Object 'Kind' is missing in 'unstructured object has no kind' --- FAIL: TestOdhOperator/components/group_2/kueue (31.05s) === RUN TestOdhOperator/components/group_2 --- FAIL: TestOdhOperator/components/group_2 (0.00s) === RUN TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable helper_test.go:150: Verifying Tenant models-as-a-service/default-tenant is present before disable helper_test.go:150: Disabling MaaS subcomponent (setting to Removed) helper_test.go:150: Waiting for Tenant models-as-a-service/default-tenant to be deleted testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable (430.18s) === RUN TestOdhOperator/components/group_4/modelsasservice === PAUSE TestOdhOperator/components/group_4/modelsasservice === CONT TestOdhOperator/components/group_4/modelsasservice 2026-07-16T13:47:31Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.20.12"}} controller_test.go:753: Creating MaaS PostgreSQL instance in namespace redhat-ods-applications controller_test.go:753: MaaS PostgreSQL instance and maas-db-config secret created in namespace redhat-ods-applications controller_test.go:753: Creating MaaS GatewayClass and Gateway: openshift-ingress/maas-default-gateway controller_test.go:753: Using hostname=maas.apps.rosa.rh-nightly-pm.nuth.p3.openshiftapps.com, certSecret=2o8clk8b6cj832lmkd2oj78crnf7qn6d-primary-cert-bundle-secret controller_test.go:753: MaaS Gateway openshift-ingress/maas-default-gateway created with HTTP+HTTPS listeners test_context_test.go:615: Timed out after 420.008s. The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1824 with: Expected resource list 'models-as-a-service/default-tenant' of kind 'Tenant' to be empty, but it contains resources. Expected <[]unstructured.Unstructured | len:1, cap:1>: [ { Object: { "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "kind": <string>"Tenant", "metadata": <map[string]interface {} | len:8>{ "resourceVersion": <string>"221305028", "uid": <string>"f46b4d94-e9bd-4e24-b8f5-d8642d6d6e03", "creationTimestamp": <string>"2026-07-16T13:48:41Z", "generation": <int64>1, "managedFields": <[]interface {} | len:2, cap:2>[ <map[string]interface {} | len:6>{ "operation": <string>"Update", "time": <string>"2026-07-16T13:48:41Z", "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:2>{ "f:metadata": <map[string]interface {} | len:1>{ "f:ownerReferences": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "k:{\"uid\":\"4febc550-d269-4e84-b8d5-2260e098eb57\"}": <map[string]interface {} | len:0>{}, }, }, "f:spec": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "f:gatewayRef": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:name": <map[string]interface {} | len:0>{}, "f:namespace": <map[string]interface {} | len:0>{}, }, }, }, "manager": <string>"manager", }, <map[string]interface {} | len:7>{ "manager": <string>"manager", "operation": <string>"Update", "subresource": <string>"status", "time": <string>"2026-07-16T13:49:05Z", "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:1>{ "f:status": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:conditions": <map[string]interface {} | len:0>{}, "f:phase": <map[string]interface {} | len:0>{}, }, }, }, ], "name": <string>"default-tenant", "namespace": <string>"models-as-a-service", "ownerReferences": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:4>{ "uid": <string>"4febc550-d269-4e84-b8d5-2260e098eb57", "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "kind": <string>"Config", "name": <string>"default", }, ], }, "spec": <map[string]interface {} | len:1>{ "gatewayRef": <map[string]interface {} | len:2>{ "name": <string>"maas-default-gateway", "namespace": <string>"openshift-ingress", }, }, "status": <map[string]interface {} | len:2>{ "conditions": <[]interface {} | len:5, cap:8>[ <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-07-16T13:48:44Z", "message": <string>"AuthConfig CRD (Kuadrant) is available on the cluster", "observedGeneration": <int64>1, "reason": <string>"DependenciesMet", "status": <string>"True", "type": <string>"DependenciesAvailable", }, <map[string]interface {} | len:6>{ "status": <string>"True", "type": <string>"MaaSPrerequisitesAvailable", "lastTransitionTime": <string>"2026-07-16T13:48:44Z", "message": <string>"All prerequisites are satisfied", "observedGeneration": <int64>1, "reason": <string>"PrerequisitesMet", }, <map[string]interface {} | len:6>{ "reason": <string>"PrerequisitesMet", "status": <string>"False", "type": <string>"Degraded", "lastTransitionTime": <string>"2026-07-16T13:48:44Z", "message": <string>"All prerequisites are satisfied", "observedGeneration": <int64>1, }, <map[string]interface {} | len:6>{ "reason": <string>"DeploymentsReady", "status": <string>"True", "type": <string>"DeploymentsAvailable", "lastTransitionTime": <string>"2026-07-16T13:49:05Z", "message": <string>"maas-api deployment is available", "observedGeneration": <int64>1, }, <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-07-16T13:49:05Z", "message": <string>"MaaS platform manifests applied and maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"Reconciled", "status": <string>"True", "type": <string>"Ready", }, ], "phase": <string>"Active", }, }, }, ] to be empty --- FAIL: TestOdhOperator/components/group_4/modelsasservice (531.60s) === RUN TestOdhOperator/components/group_4 --- FAIL: TestOdhOperator/components/group_4 (0.00s) === RUN TestOdhOperator/components 2026/07/16 13:43:21 TEST FAILURE DETECTED: group 2 2026/07/16 13:43:21 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 2) === 2026/07/16 13:43:22 === CLUSTER STATE === 2026/07/16 13:43:22 Node: ip-10-0-1-132.us-east-2.compute.internal 2026/07/16 13:43:22 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 13:43:22 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 13:43:22 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 13:43:22 Condition Ready: True - kubelet is posting ready status 2026/07/16 13:43:22 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/16 13:43:22 Node: ip-10-0-1-189.us-east-2.compute.internal 2026/07/16 13:43:22 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 13:43:22 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 13:43:22 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 13:43:22 Condition Ready: True - kubelet is posting ready status 2026/07/16 13:43:22 Resources: 7500m CPU, 30892176Ki memory allocatable (of 8000m CPU, 32043152Ki memory capacity) 2026/07/16 13:43:22 Node: ip-10-0-1-39.us-east-2.compute.internal 2026/07/16 13:43:22 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 13:43:22 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 13:43:22 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 13:43:22 Condition Ready: True - kubelet is posting ready status 2026/07/16 13:43:22 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/16 13:43:22 === DEPLOYMENTS === 2026/07/16 13:43:22 Namespace: redhat-ods-applications 2026/07/16 13:43:22 Deployment maas-api: 1/1 (READY) 2026/07/16 13:43:22 Deployment maas-controller: 1/1 (READY) 2026/07/16 13:43:22 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/16 13:43:22 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/16 13:43:22 Namespace: kube-system 2026/07/16 13:43:22 === PODS === 2026/07/16 13:43:22 Namespace redhat-ods-applications: no problematic pods found 2026/07/16 13:43:22 Namespace: kube-system 2026/07/16 13:43:22 Pod ecr-credential-refresh-initial-k6gr5: Succeeded 2026/07/16 13:43:22 Container refresh: ready=false, terminated: Completed (exit 0) 2026/07/16 13:43:22 === LOGS (CURRENT) for container refresh === 2026/07/16 13:43:22 2026-04-23 02:42:29 - Discovering ECR registries from cluster configuration... 2026/07/16 13:43:22 2026-04-23 02:42:31 - No IDMS mirrors found 2026/07/16 13:43:22 2026-04-23 02:42:31 - Skipping - no ECR IDMS mirrors found (not a zero-egress cluster) 2026/07/16 13:43:22 === END LOGS === 2026/07/16 13:43:22 === OPERATOR STATUS === 2026/07/16 13:43:22 Operator deployment: 3/3 ready 2026/07/16 13:43:22 === DSCI STATUS === 2026/07/16 13:43:22 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/16 13:43:22 === DSC STATUS === 2026/07/16 13:43:22 Failed to collect DSC data: default-dsc unhealthy conditions: ModelsAsServiceReady=False 2026/07/16 13:43:22 === RECENT EVENTS === 2026/07/16 13:43:22 13:43:15 INFO Pod/model-registry-operator-controller-manager-66d8c86b58-t4p2c: Killing - Stopping container manager 2026/07/16 13:43:22 13:43:15 INFO Pod/model-serving-api-6688cd56bd-927mp: Killing - Stopping container server 2026/07/16 13:43:22 13:43:15 INFO Pod/llmisvc-controller-manager-5d65df97bb-tjjq2: Killing - Stopping container manager 2026/07/16 13:43:22 13:43:15 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-ffbm5: Killing - Stopping container manager 2026/07/16 13:43:22 13:43:15 INFO Pod/odh-model-controller-867855946f-zp662: Killing - Stopping container manager 2026/07/16 13:43:22 13:43:15 INFO Pod/kserve-controller-manager-6c67dcb69c-wbpc4: Killing - Stopping container manager 2026/07/16 13:43:22 13:43:12 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 13:43:22 13:43:12 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 13:43:22 13:42:58 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-867855946f-zp662_6e49c27d-3715-424a-9a2c-cac023a66062 became leader 2026/07/16 13:43:22 13:42:57 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-ffbm5: Created - Created container: manager 2026/07/16 13:43:22 13:42:57 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-ffbm5: Started - Started container manager 2026/07/16 13:43:22 13:42:57 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-5bd688895b-ffbm5_504fe41a-0718-4ceb-9ec5-990fb0761314 became leader 2026/07/16 13:43:22 13:42:57 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-ffbm5: Pulled - Container image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" already present on machine 2026/07/16 13:43:22 13:42:57 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-ffbm5: AddedInterface - Add eth0 [10.130.1.146/23] from ovn-kubernetes 2026/07/16 13:43:22 13:42:57 WARN Pod/workload-variant-autoscaler-controller-manager-5bd688895b-fd6mz: Unhealthy - Readiness probe failed: Get "http://10.130.1.140:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026/07/16 13:43:22 13:42:56 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-ffbm5: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-5bd688895b-ffbm5 to ip-10-0-1-189.us-east-2.compute.internal 2026/07/16 13:43:22 13:42:56 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-5bd688895b from 0 to 1 2026/07/16 13:43:22 13:42:56 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-5bd688895b-ffbm5 2026/07/16 13:43:22 13:42:56 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-pgkkb: Killing - Stopping container manager 2026/07/16 13:43:22 13:42:56 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-fd6mz: Killing - Stopping container manager 2026/07/16 13:43:22 13:42:42 INFO Pod/odh-model-controller-867855946f-zp662: Started - Started container manager 2026/07/16 13:43:22 13:42:42 INFO Pod/odh-model-controller-867855946f-zp662: Created - Created container: manager 2026/07/16 13:43:22 13:42:42 INFO Pod/odh-model-controller-867855946f-zp662: AddedInterface - Add eth0 [10.130.1.145/23] from ovn-kubernetes 2026/07/16 13:43:22 13:42:42 INFO Pod/odh-model-controller-867855946f-zp662: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/16 13:43:22 13:42:42 INFO Pod/odh-model-controller-867855946f-zp662: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 653ms (653ms including waiting). Image size: 238959091 bytes. 2026/07/16 13:43:22 13:42:41 INFO Pod/odh-model-controller-867855946f-zp662: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-867855946f-zp662 to ip-10-0-1-189.us-east-2.compute.internal 2026/07/16 13:43:22 13:42:41 INFO Pod/odh-model-controller-867855946f-tk82d: Killing - Stopping container manager 2026/07/16 13:43:22 13:42:41 INFO ReplicaSet/odh-model-controller-867855946f: SuccessfulCreate - Created pod: odh-model-controller-867855946f-zp662 2026/07/16 13:43:22 13:42:41 INFO Pod/odh-model-controller-867855946f-b5l64: Killing - Stopping container manager 2026/07/16 13:43:22 13:42:41 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-867855946f from 0 to 1 2026/07/16 13:43:22 13:42:27 INFO Pod/model-serving-api-6688cd56bd-927mp: Created - Created container: server 2026/07/16 13:43:22 13:42:27 INFO Pod/model-serving-api-6688cd56bd-927mp: Started - Started container server 2026/07/16 13:43:22 13:42:27 INFO Pod/model-serving-api-6688cd56bd-927mp: AddedInterface - Add eth0 [10.130.1.144/23] from ovn-kubernetes 2026/07/16 13:43:22 13:42:27 INFO Pod/model-serving-api-6688cd56bd-927mp: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/16 13:43:22 13:42:26 INFO Pod/model-serving-api-6688cd56bd-927mp: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-927mp to ip-10-0-1-189.us-east-2.compute.internal 2026/07/16 13:43:22 13:42:26 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-927mp 2026/07/16 13:43:22 13:42:26 INFO Pod/model-serving-api-6688cd56bd-wmxg9: Killing - Stopping container server 2026/07/16 13:43:22 13:42:26 INFO Pod/model-serving-api-6688cd56bd-b5lqh: Killing - Stopping container server 2026/07/16 13:43:22 13:42:26 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/16 13:43:22 13:41:26 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-pgkkb: Created - Created container: manager 2026/07/16 13:43:22 13:41:26 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-pgkkb: AddedInterface - Add eth0 [10.129.70.170/23] from ovn-kubernetes 2026/07/16 13:43:22 13:41:26 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-pgkkb: Pulled - Container image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" already present on machine 2026/07/16 13:43:22 13:41:26 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-pgkkb: Started - Started container manager 2026/07/16 13:43:22 13:41:25 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-pgkkb: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-5bd688895b-pgkkb to ip-10-0-1-132.us-east-2.compute.internal 2026/07/16 13:43:22 13:41:25 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-5bd688895b-pgkkb 2026/07/16 13:43:22 13:41:25 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-5bd688895b from 1 to 2 2026/07/16 13:43:22 13:41:06 INFO Pod/odh-model-controller-867855946f-b5l64: AddedInterface - Add eth0 [10.129.70.169/23] from ovn-kubernetes 2026/07/16 13:43:22 13:41:06 INFO Pod/odh-model-controller-867855946f-b5l64: Created - Created container: manager 2026/07/16 13:43:22 13:41:06 INFO Pod/odh-model-controller-867855946f-b5l64: Started - Started container manager 2026/07/16 13:43:22 13:41:06 INFO Pod/odh-model-controller-867855946f-b5l64: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 690ms (690ms including waiting). Image size: 238959091 bytes. 2026/07/16 13:43:22 13:41:06 INFO Pod/odh-model-controller-867855946f-b5l64: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/16 13:43:22 13:41:05 INFO Pod/odh-model-controller-867855946f-b5l64: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-867855946f-b5l64 to ip-10-0-1-132.us-east-2.compute.internal 2026/07/16 13:43:22 13:41:05 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-867855946f from 1 to 2 2026/07/16 13:43:22 13:41:05 INFO ReplicaSet/odh-model-controller-867855946f: SuccessfulCreate - Created pod: odh-model-controller-867855946f-b5l64 2026/07/16 13:43:22 13:40:46 INFO Pod/model-serving-api-6688cd56bd-wmxg9: AddedInterface - Add eth0 [10.129.70.168/23] from ovn-kubernetes 2026/07/16 13:43:22 13:40:46 INFO Pod/model-serving-api-6688cd56bd-wmxg9: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/16 13:43:22 13:40:46 INFO Pod/model-serving-api-6688cd56bd-wmxg9: Created - Created container: server 2026/07/16 13:43:22 13:40:46 INFO Pod/model-serving-api-6688cd56bd-wmxg9: Started - Started container server 2026/07/16 13:43:22 13:40:45 INFO Pod/model-serving-api-6688cd56bd-wmxg9: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-wmxg9 to ip-10-0-1-132.us-east-2.compute.internal 2026/07/16 13:43:22 13:40:45 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-wmxg9 2026/07/16 13:43:22 13:40:45 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 1 to 2 2026/07/16 13:43:22 13:40:37 INFO Pod/maas-api-84449bdd5d-8mdsc: Created - Created container: maas-api 2026/07/16 13:43:22 13:40:37 INFO Pod/maas-api-84449bdd5d-8mdsc: Started - Started container maas-api 2026/07/16 13:43:22 13:40:37 INFO Pod/maas-api-84449bdd5d-8mdsc: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" in 2.279s (2.279s including waiting). Image size: 175018396 bytes. 2026/07/16 13:43:22 13:40:35 INFO Pod/maas-api-84449bdd5d-8mdsc: Pulling - Pulling image "quay.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" 2026/07/16 13:43:22 13:40:35 INFO Pod/maas-api-84449bdd5d-8mdsc: AddedInterface - Add eth0 [10.129.70.165/23] from ovn-kubernetes 2026/07/16 13:43:22 13:40:34 INFO Pod/maas-api-84449bdd5d-8mdsc: Scheduled - Successfully assigned redhat-ods-applications/maas-api-84449bdd5d-8mdsc to ip-10-0-1-132.us-east-2.compute.internal 2026/07/16 13:43:22 13:40:34 WARN ReplicaSet/maas-api-84449bdd5d: FailedCreate - Error creating: pods "maas-api-84449bdd5d-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 2026/07/16 13:43:22 13:40:34 INFO ReplicaSet/maas-api-84449bdd5d: SuccessfulCreate - Created pod: maas-api-84449bdd5d-8mdsc 2026/07/16 13:43:22 13:40:34 INFO Deployment/maas-api: ScalingReplicaSet - Scaled up replica set maas-api-84449bdd5d from 0 to 1 2026/07/16 13:43:22 13:40:19 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-tjjq2_6998e678-107b-4dce-9bb8-2c48783adbe7 became leader 2026/07/16 13:43:22 13:40:18 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-wbpc4_7b2be1ff-993f-4327-a782-08e0c95821de became leader 2026/07/16 13:43:22 13:40:05 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-fd6mz: Created - Created container: manager 2026/07/16 13:43:22 13:40:05 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-5bd688895b-fd6mz_147985d1-f3af-44ff-bd4c-c2d559be7aae became leader 2026/07/16 13:43:22 13:40:05 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-fd6mz: Pulled - Container image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" already present on machine 2026/07/16 13:43:22 13:40:05 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-fd6mz: Started - Started container manager 2026/07/16 13:43:22 13:40:05 WARN ServiceMonitor/workload-variant-autoscaler-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/workload-variant-autoscaler-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: bearerTokenSecret: [REDACTED] to get secret "workload-variant-autoscaler-controller-manager-token": secrets "workload-variant-autoscaler-controller-manager-token" not found 2026/07/16 13:43:22 13:40:05 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-fd6mz: AddedInterface - Add eth0 [10.130.1.140/23] from ovn-kubernetes 2026/07/16 13:43:22 13:40:04 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-fd6mz: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-5bd688895b-fd6mz to ip-10-0-1-189.us-east-2.compute.internal 2026/07/16 13:43:22 13:40:04 WARN Pod/workload-variant-autoscaler-controller-manager-5bd688895b-fd6mz: FailedMount - MountVolume.SetUp failed for volume "epp-metrics-token" : secret "workload-variant-autoscaler-epp-metrics-token" not found 2026/07/16 13:43:22 13:40:04 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-5bd688895b-fd6mz 2026/07/16 13:43:22 13:40:03 WARN ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: FailedCreate - Error creating: pods "workload-variant-autoscaler-controller-manager-5bd688895b-" is forbidden: error looking up service account redhat-ods-applications/workload-variant-autoscaler-controller-manager: serviceaccount "workload-variant-autoscaler-controller-manager" not found 2026/07/16 13:43:22 13:40:03 INFO Lease/85f368d1.opendatahub.io: LeaderElection - model-registry-operator-controller-manager-66d8c86b58-t4p2c_0081330f-bb09-493d-9817-012020ce5539 became leader 2026/07/16 13:43:22 13:40:03 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-5bd688895b from 0 to 1 2026/07/16 13:43:22 13:40:02 INFO Pod/kserve-controller-manager-6c67dcb69c-wbpc4: Started - Started container manager 2026/07/16 13:43:22 13:40:02 INFO Lease/maas-controller.models-as-a-service.opendatahub.io: LeaderElection - maas-controller-85bdcf9988-t4j68_5af5332c-1a61-45bd-aced-87740872bcfa became leader 2026/07/16 13:43:22 13:40:02 INFO Pod/kserve-controller-manager-6c67dcb69c-wbpc4: Created - Created container: manager 2026/07/16 13:43:22 13:40:02 INFO Pod/kserve-controller-manager-6c67dcb69c-wbpc4: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 2.406s (2.406s including waiting). Image size: 238435613 bytes. 2026/07/16 13:43:22 13:40:02 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-867855946f-tk82d_f25b3c95-49ef-4092-87cd-b0dccaca7557 became leader 2026/07/16 13:43:22 13:40:01 INFO Pod/llmisvc-controller-manager-5d65df97bb-tjjq2: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 904ms (904ms including waiting). Image size: 236366599 bytes. 2026/07/16 13:43:22 13:40:01 INFO Pod/llmisvc-controller-manager-5d65df97bb-tjjq2: Started - Started container manager 2026/07/16 13:43:22 13:40:01 INFO Pod/llmisvc-controller-manager-5d65df97bb-tjjq2: Created - Created container: manager 2026/07/16 13:43:22 13:40:00 INFO Pod/llmisvc-controller-manager-5d65df97bb-tjjq2: AddedInterface - Add eth0 [10.129.70.162/23] from ovn-kubernetes 2026/07/16 13:43:22 13:40:00 INFO Pod/llmisvc-controller-manager-5d65df97bb-tjjq2: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/16 13:43:22 13:40:00 INFO Pod/kserve-controller-manager-6c67dcb69c-wbpc4: AddedInterface - Add eth0 [10.130.1.138/23] from ovn-kubernetes 2026/07/16 13:43:22 13:40:00 INFO Pod/kserve-controller-manager-6c67dcb69c-wbpc4: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/16 13:43:22 13:39:59 INFO Pod/llmisvc-controller-manager-5d65df97bb-tjjq2: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-tjjq2 to ip-10-0-1-132.us-east-2.compute.internal 2026/07/16 13:43:22 13:39:59 INFO Pod/kserve-controller-manager-6c67dcb69c-wbpc4: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-wbpc4 to ip-10-0-1-189.us-east-2.compute.internal 2026/07/16 13:43:22 13:39:59 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-tjjq2 2026/07/16 13:43:22 13:39:59 WARN Pod/llmisvc-controller-manager-5d65df97bb-tjjq2: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 2026/07/16 13:43:22 13:39:59 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/16 13:43:22 13:39:59 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/16 13:43:22 13:39:59 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-wbpc4 2026/07/16 13:43:22 13:39:59 WARN Pod/kserve-controller-manager-6c67dcb69c-wbpc4: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 2026/07/16 13:43:22 13:39:48 INFO Pod/model-registry-operator-controller-manager-66d8c86b58-t4p2c: Created - Created container: manager 2026/07/16 13:43:22 13:39:48 INFO Pod/model-registry-operator-controller-manager-66d8c86b58-t4p2c: Started - Started container manager 2026/07/16 13:43:22 13:39:47 INFO Pod/model-registry-operator-controller-manager-66d8c86b58-t4p2c: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-registry-operator-rhel9@sha256:ebcec5c43468283ed3cbab50530f939166f8f3e8c9d306e91a8107d05d7fa9d4" in 4.635s (4.635s including waiting). Image size: 211356146 bytes. 2026/07/16 13:43:22 13:39:45 INFO Pod/model-serving-api-6688cd56bd-b5lqh: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/16 13:43:22 13:39:45 INFO Pod/maas-controller-85bdcf9988-t4j68: Started - Started container manager 2026/07/16 13:43:22 13:39:45 INFO Pod/maas-controller-85bdcf9988-t4j68: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 2.929s (2.929s including waiting). Image size: 212866077 bytes. 2026/07/16 13:43:22 13:39:45 INFO Pod/model-serving-api-6688cd56bd-b5lqh: Created - Created container: server 2026/07/16 13:43:22 13:39:45 INFO Pod/model-serving-api-6688cd56bd-b5lqh: Started - Started container server 2026/07/16 13:43:22 13:39:45 INFO Pod/maas-controller-85bdcf9988-t4j68: Created - Created container: manager 2026/07/16 13:43:22 13:39:45 INFO Pod/model-serving-api-6688cd56bd-b5lqh: AddedInterface - Add eth0 [10.130.1.137/23] from ovn-kubernetes 2026/07/16 13:43:22 13:39:44 INFO Pod/model-serving-api-6688cd56bd-b5lqh: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-b5lqh to ip-10-0-1-189.us-east-2.compute.internal 2026/07/16 13:43:22 13:39:44 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-b5lqh 2026/07/16 13:43:22 13:39:44 INFO Pod/odh-model-controller-867855946f-tk82d: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 595ms (595ms including waiting). Image size: 238959091 bytes. 2026/07/16 13:43:22 13:39:44 INFO Pod/odh-model-controller-867855946f-tk82d: Started - Started container manager 2026/07/16 13:43:22 13:39:44 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/16 13:43:22 13:39:44 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/16 13:43:22 13:39:44 INFO Pod/odh-model-controller-867855946f-tk82d: Created - Created container: manager 2026/07/16 13:43:22 13:39:43 INFO Pod/model-registry-operator-controller-manager-66d8c86b58-t4p2c: AddedInterface - Add eth0 [10.130.1.136/23] from ovn-kubernetes 2026/07/16 13:43:22 13:39:43 INFO Pod/model-registry-operator-controller-manager-66d8c86b58-t4p2c: Pulling - Pulling image "quay.io/rhoai/odh-model-registry-operator-rhel9@sha256:ebcec5c43468283ed3cbab50530f939166f8f3e8c9d306e91a8107d05d7fa9d4" 2026/07/16 13:43:22 13:39:43 INFO Pod/odh-model-controller-867855946f-tk82d: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/16 13:43:22 13:39:43 INFO Pod/odh-model-controller-867855946f-tk82d: AddedInterface - Add eth0 [10.130.1.135/23] from ovn-kubernetes 2026/07/16 13:43:22 13:39:42 INFO Pod/model-registry-operator-controller-manager-66d8c86b58-t4p2c: Scheduled - Successfully assigned redhat-ods-applications/model-registry-operator-controller-manager-66d8c86b58-t4p2c to ip-10-0-1-189.us-east-2.compute.internal 2026/07/16 13:43:22 13:39:42 INFO ReplicaSet/model-registry-operator-controller-manager-66d8c86b58: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-66d8c86b58-t4p2c 2026/07/16 13:43:22 13:39:42 INFO Pod/maas-controller-85bdcf9988-t4j68: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/16 13:43:22 13:39:42 WARN Pod/odh-model-controller-867855946f-tk82d: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/16 13:43:22 13:39:42 WARN Pod/model-registry-operator-controller-manager-66d8c86b58-t4p2c: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 2026/07/16 13:43:22 13:39:42 INFO Pod/maas-controller-85bdcf9988-t4j68: AddedInterface - Add eth0 [10.130.1.134/23] from ovn-kubernetes 2026/07/16 13:43:22 13:39:42 INFO Deployment/model-registry-operator-controller-manager: ScalingReplicaSet - Scaled up replica set model-registry-operator-controller-manager-66d8c86b58 from 0 to 1 2026/07/16 13:43:22 13:39:41 INFO Pod/odh-model-controller-867855946f-tk82d: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-867855946f-tk82d to ip-10-0-1-189.us-east-2.compute.internal 2026/07/16 13:43:22 13:39:41 INFO Pod/maas-controller-85bdcf9988-t4j68: Scheduled - Successfully assigned redhat-ods-applications/maas-controller-85bdcf9988-t4j68 to ip-10-0-1-189.us-east-2.compute.internal 2026/07/16 13:43:22 13:39:41 INFO Deployment/maas-controller: ScalingReplicaSet - Scaled up replica set maas-controller-85bdcf9988 from 0 to 1 2026/07/16 13:43:22 13:39:41 INFO ReplicaSet/maas-controller-85bdcf9988: SuccessfulCreate - Created pod: maas-controller-85bdcf9988-t4j68 2026/07/16 13:43:22 13:39:41 INFO ReplicaSet/odh-model-controller-867855946f: SuccessfulCreate - Created pod: odh-model-controller-867855946f-tk82d 2026/07/16 13:43:22 13:39:41 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-867855946f from 0 to 1 2026/07/16 13:43:22 13:39:32 INFO Pod/dashboard-redirect-85dbcf645f-xd8dk: Killing - Stopping container nginx 2026/07/16 13:43:22 13:39:32 INFO Pod/dashboard-redirect-85dbcf645f-68pfs: Killing - Stopping container nginx 2026/07/16 13:43:22 13:38:41 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" in 453ms (453ms including waiting). Image size: 185092464 bytes. 2026/07/16 13:43:22 13:38:41 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" 2026/07/16 13:43:22 13:38:41 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: Started - Started container mlflow-ui 2026/07/16 13:43:22 13:38:41 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: Created - Created container: mlflow-ui 2026/07/16 13:43:22 13:38:41 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" in 528ms (528ms including waiting). Image size: 177808591 bytes. 2026/07/16 13:43:22 13:38:41 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" in 2.456s (2.456s including waiting). Image size: 185092464 bytes. 2026/07/16 13:43:22 13:38:40 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: Created - Created container: maas-ui 2026/07/16 13:43:22 13:38:40 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: Started - Started container maas-ui 2026/07/16 13:43:22 13:38:40 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" in 491ms (491ms including waiting). Image size: 187802205 bytes. 2026/07/16 13:43:22 13:38:40 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" 2026/07/16 13:43:22 13:38:40 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: Started - Started container gen-ai-ui 2026/07/16 13:43:22 13:38:40 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: Created - Created container: gen-ai-ui 2026/07/16 13:43:22 13:38:40 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" in 2.197s (2.197s including waiting). Image size: 241242949 bytes. 2026/07/16 13:43:22 13:38:40 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" 2026/07/16 13:43:22 13:38:39 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: Started - Started container mlflow-ui 2026/07/16 13:43:22 13:38:39 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: Created - Created container: mlflow-ui 2026/07/16 13:43:22 13:38:39 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" 2026/07/16 13:43:22 13:38:39 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" in 490ms (490ms including waiting). Image size: 177808591 bytes. 2026/07/16 13:43:22 13:38:38 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" in 627ms (627ms including waiting). Image size: 187802205 bytes. 2026/07/16 13:43:22 13:38:38 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: Created - Created container: maas-ui 2026/07/16 13:43:22 13:38:38 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: Started - Started container maas-ui 2026/07/16 13:43:22 13:38:38 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" 2026/07/16 13:43:22 13:38:37 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: Started - Started container model-registry-ui 2026/07/16 13:43:22 13:38:37 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" 2026/07/16 13:43:22 13:38:37 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" 2026/07/16 13:43:22 13:38:37 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" in 540ms (540ms including waiting). Image size: 241242949 bytes. 2026/07/16 13:43:22 13:38:37 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" 2026/07/16 13:43:22 13:38:37 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: Started - Started container gen-ai-ui 2026/07/16 13:43:22 13:38:37 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: Created - Created container: model-registry-ui 2026/07/16 13:43:22 13:38:37 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" in 681ms (681ms including waiting). Image size: 182438697 bytes. 2026/07/16 13:43:22 13:38:37 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" 2026/07/16 13:43:22 13:38:37 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: Created - Created container: gen-ai-ui 2026/07/16 13:43:22 13:38:37 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: Started - Started container kube-rbac-proxy 2026/07/16 13:43:22 13:38:37 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: Created - Created container: kube-rbac-proxy 2026/07/16 13:43:22 13:38:37 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: Pulled - Container image "quay.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:1e614e46799fd020ac18d95c06a463421154ce86e59ccb3fa6007e66fed9910e" already present on machine 2026/07/16 13:43:22 13:38:37 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: Started - Started container rhods-dashboard 2026/07/16 13:43:22 13:38:37 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: Created - Created container: rhods-dashboard 2026/07/16 13:43:22 13:38:37 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: Started - Started container model-registry-ui 2026/07/16 13:43:22 13:38:37 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: Created - Created container: model-registry-ui 2026/07/16 13:43:22 13:38:37 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" in 608ms (608ms including waiting). Image size: 182438697 bytes. 2026/07/16 13:43:22 13:38:36 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: AddedInterface - Add eth0 [10.129.70.161/23] from ovn-kubernetes 2026/07/16 13:43:22 13:38:36 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: Started - Started container rhods-dashboard 2026/07/16 13:43:22 13:38:36 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: Pulling - Pulling image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" 2026/07/16 13:43:22 13:38:36 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: Pulled - Successfully pulled image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" in 838ms (838ms including waiting). Image size: 799822033 bytes. 2026/07/16 13:43:22 13:38:36 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: Pulled - Successfully pulled image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" in 620ms (620ms including waiting). Image size: 799822033 bytes. 2026/07/16 13:43:22 13:38:36 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: Created - Created container: rhods-dashboard 2026/07/16 13:43:22 13:38:36 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: Pulled - Container image "quay.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:1e614e46799fd020ac18d95c06a463421154ce86e59ccb3fa6007e66fed9910e" already present on machine 2026/07/16 13:43:22 13:38:36 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: Created - Created container: kube-rbac-proxy 2026/07/16 13:43:22 13:38:36 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: Started - Started container kube-rbac-proxy 2026/07/16 13:43:22 13:38:36 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" 2026/07/16 13:43:22 13:38:35 INFO Pod/rhods-dashboard-5998574cd9-jjpfm: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-5998574cd9-jjpfm to ip-10-0-1-132.us-east-2.compute.internal 2026/07/16 13:43:22 13:38:35 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-5998574cd9-mmkwq to ip-10-0-1-189.us-east-2.compute.internal 2026/07/16 13:43:22 13:38:35 INFO ReplicaSet/rhods-dashboard-5998574cd9: SuccessfulCreate - Created pod: rhods-dashboard-5998574cd9-mmkwq 2026/07/16 13:43:22 13:38:35 INFO ReplicaSet/rhods-dashboard-5998574cd9: SuccessfulCreate - Created pod: rhods-dashboard-5998574cd9-jjpfm 2026/07/16 13:43:22 13:38:35 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: Pulling - Pulling image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" 2026/07/16 13:43:22 13:38:35 INFO Pod/rhods-dashboard-5998574cd9-mmkwq: AddedInterface - Add eth0 [10.130.1.133/23] from ovn-kubernetes 2026/07/16 13:43:22 13:38:34 INFO Deployment/rhods-dashboard: ScalingReplicaSet - Scaled up replica set rhods-dashboard-5998574cd9 from 0 to 2 2026/07/16 13:43:22 13:38:33 INFO Pod/spark-operator-controller-767899d44f-lsjxk: Killing - Stopping container controller 2026/07/16 13:43:22 13:38:33 INFO Pod/spark-operator-webhook-9cb9bdf44-4jvw4: Killing - Stopping container webhook 2026/07/16 13:43:22 === RESOURCE QUOTAS === 2026/07/16 13:43:22 No resource quotas found in monitored namespaces 2026/07/16 13:43:22 Classification for group 2: infrastructure/storage (code=1006, confidence=medium) 2026/07/16 13:43:22 Evidence: event Pod/workload-variant-autoscaler-controller-manager-5bd688895b-fd6mz: FailedMount - MountVolume.SetUp failed for volume "epp-metrics-token" : secret "workload-variant-autoscaler-epp-metrics-token" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/workload-variant-autoscaler-controller-manager-5bd688895b-fd6mz: FailedMount - MountVolume.SetUp failed for volume \"epp-metrics-token\" : secret \"workload-variant-autoscaler-epp-metrics-token\" not found"],"confidence":"medium"} 2026/07/16 13:43:22 Diagnostics complete for failed test: group 2 controller_test.go:753: Stopping: group 2 test failed. 2026/07/16 13:43:22 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/16 13:43:22 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/16 13:47:31 Circuit breaker: success recorded, resetting failure counter (was 1) 2026/07/16 13:56:23 TEST FAILURE DETECTED: group 4 2026/07/16 13:56:23 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 4) === 2026/07/16 13:56:23 === CLUSTER STATE === 2026/07/16 13:56:23 Node: ip-10-0-1-132.us-east-2.compute.internal 2026/07/16 13:56:23 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 13:56:23 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 13:56:23 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 13:56:23 Condition Ready: True - kubelet is posting ready status 2026/07/16 13:56:23 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/16 13:56:23 Node: ip-10-0-1-189.us-east-2.compute.internal 2026/07/16 13:56:23 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 13:56:23 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 13:56:23 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 13:56:23 Condition Ready: True - kubelet is posting ready status 2026/07/16 13:56:23 Resources: 7500m CPU, 30892176Ki memory allocatable (of 8000m CPU, 32043152Ki memory capacity) 2026/07/16 13:56:23 Node: ip-10-0-1-39.us-east-2.compute.internal 2026/07/16 13:56:23 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 13:56:23 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 13:56:23 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 13:56:23 Condition Ready: True - kubelet is posting ready status 2026/07/16 13:56:23 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/16 13:56:23 === DEPLOYMENTS === 2026/07/16 13:56:23 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/16 13:56:23 === PODS === 2026/07/16 13:56:23 Failed to collect pod data: redhat-ods-applications/maas-controller-85bdcf9988-d6l59: container manager not ready 2026/07/16 13:56:23 === OPERATOR STATUS === 2026/07/16 13:56:23 Operator deployment: 3/3 ready 2026/07/16 13:56:23 === DSCI STATUS === 2026/07/16 13:56:23 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/16 13:56:23 === DSC STATUS === 2026/07/16 13:56:23 DSC default-dsc: 2026/07/16 13:56:23 Ready: True - 2026/07/16 13:56:23 ProvisioningSucceeded: True - 2026/07/16 13:56:23 ComponentsReady: True - 2026/07/16 13:56:23 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/16 13:56:23 DashboardReady: False - Component ManagementState is set to Removed 2026/07/16 13:56:23 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/16 13:56:23 KserveLLMInferenceServiceDependencies: True - 2026/07/16 13:56:23 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/16 13:56:23 KserveReady: True - 2026/07/16 13:56:23 KueueReady: True - 2026/07/16 13:56:23 LLM-D-WVADependencies: True - 2026/07/16 13:56:23 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/16 13:56:23 ModelControllerReady: True - 2026/07/16 13:56:23 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/16 13:56:23 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/16 13:56:23 OGXReady: False - Component ManagementState is set to Removed 2026/07/16 13:56:23 RayReady: False - Component ManagementState is set to Removed 2026/07/16 13:56:23 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/16 13:56:23 TrainerReady: False - Component ManagementState is set to Removed 2026/07/16 13:56:23 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/16 13:56:23 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/16 13:56:23 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/16 13:56:23 === RECENT EVENTS === 2026/07/16 13:56:23 13:55:13 INFO Pod/maas-controller-85bdcf9988-d6l59: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 647ms (647ms including waiting). Image size: 212866077 bytes. 2026/07/16 13:56:23 13:55:12 INFO Pod/maas-controller-85bdcf9988-d6l59: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/16 13:56:23 13:54:17 WARN Pod/maas-controller-85bdcf9988-d6l59: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-94f49); attempting to pull the image may not succeed. 2026/07/16 13:56:23 13:53:24 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 13:56:23 13:52:22 INFO Pod/maas-controller-85bdcf9988-d6l59: Created - Created container: manager 2026/07/16 13:56:23 13:52:22 INFO Pod/maas-controller-85bdcf9988-d6l59: Started - Started container manager 2026/07/16 13:56:23 13:52:22 INFO Pod/maas-controller-85bdcf9988-d6l59: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 2.669s (2.669s including waiting). Image size: 212866077 bytes. 2026/07/16 13:56:23 === RESOURCE QUOTAS === 2026/07/16 13:56:23 No resource quotas found in monitored namespaces 2026/07/16 13:56:23 Classification for group 4: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/16 13:56:23 Evidence: container maas-controller-85bdcf9988-d6l59/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-d6l59_redhat-ods-applications(b5835c58-4c0c-4fbf-ad57-98396b8258ee) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-85bdcf9988-d6l59/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-d6l59_redhat-ods-applications(b5835c58-4c0c-4fbf-ad57-98396b8258ee)"],"confidence":"medium"} 2026/07/16 13:56:23 Diagnostics complete for failed test: group 4 controller_test.go:753: Stopping: group 4 test failed. 2026/07/16 13:56:23 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/16 13:56:23 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/16 13:57:34 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (1486.27s) === RUN TestOdhOperator 2026/07/16 13:33:38 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/16 13:58:35 TEST FAILURE DETECTED: components 2026/07/16 13:58:35 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/16 13:58:36 === CLUSTER STATE === 2026/07/16 13:58:36 Node: ip-10-0-1-132.us-east-2.compute.internal 2026/07/16 13:58:36 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 13:58:36 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 13:58:36 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 13:58:36 Condition Ready: True - kubelet is posting ready status 2026/07/16 13:58:36 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/16 13:58:36 Node: ip-10-0-1-189.us-east-2.compute.internal 2026/07/16 13:58:36 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 13:58:36 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 13:58:36 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 13:58:36 Condition Ready: True - kubelet is posting ready status 2026/07/16 13:58:36 Resources: 7500m CPU, 30892176Ki memory allocatable (of 8000m CPU, 32043152Ki memory capacity) 2026/07/16 13:58:36 Node: ip-10-0-1-39.us-east-2.compute.internal 2026/07/16 13:58:36 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 13:58:36 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 13:58:36 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 13:58:36 Condition Ready: True - kubelet is posting ready status 2026/07/16 13:58:36 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/16 13:58:36 === DEPLOYMENTS === 2026/07/16 13:58:36 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/16 13:58:36 === PODS === 2026/07/16 13:58:36 Failed to collect pod data: redhat-ods-applications/maas-controller-85bdcf9988-d6l59: container manager not ready 2026/07/16 13:58:36 === OPERATOR STATUS === 2026/07/16 13:58:36 Operator deployment: 3/3 ready 2026/07/16 13:58:36 === DSCI STATUS === 2026/07/16 13:58:36 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/16 13:58:36 === DSC STATUS === 2026/07/16 13:58:36 DSC default-dsc: 2026/07/16 13:58:36 Ready: True - 2026/07/16 13:58:36 ProvisioningSucceeded: True - 2026/07/16 13:58:36 ComponentsReady: True - 2026/07/16 13:58:36 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/16 13:58:36 DashboardReady: False - Component ManagementState is set to Removed 2026/07/16 13:58:36 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/16 13:58:36 KserveLLMInferenceServiceDependencies: True - 2026/07/16 13:58:36 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/16 13:58:36 KserveReady: True - 2026/07/16 13:58:36 KueueReady: True - 2026/07/16 13:58:36 LLM-D-WVADependencies: True - 2026/07/16 13:58:36 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/16 13:58:36 ModelControllerReady: True - 2026/07/16 13:58:36 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/16 13:58:36 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/16 13:58:36 OGXReady: False - Component ManagementState is set to Removed 2026/07/16 13:58:36 RayReady: False - Component ManagementState is set to Removed 2026/07/16 13:58:36 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/16 13:58:36 TrainerReady: True - 2026/07/16 13:58:36 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/16 13:58:36 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/16 13:58:36 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/16 13:58:36 === RECENT EVENTS === 2026/07/16 13:58:36 13:58:19 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-7ftqw_3d1d8cf1-a18e-435e-b54e-067374ab585f became leader 2026/07/16 13:58:36 13:58:19 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-nskhp_c776e279-81a3-4400-8d3c-6299325b8b6d became leader 2026/07/16 13:58:36 13:58:04 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-f2q2t_2ef5ca66-70d2-4cda-882e-ce71fdfd223c became leader 2026/07/16 13:58:36 13:58:02 INFO Pod/llmisvc-controller-manager-5d65df97bb-7ftqw: Created - Created container: manager 2026/07/16 13:58:36 13:58:02 INFO Pod/llmisvc-controller-manager-5d65df97bb-7ftqw: Started - Started container manager 2026/07/16 13:58:36 13:58:01 INFO Pod/kserve-controller-manager-6c67dcb69c-nskhp: Started - Started container manager 2026/07/16 13:58:36 13:58:01 INFO Pod/llmisvc-controller-manager-5d65df97bb-7ftqw: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 790ms (790ms including waiting). Image size: 236366599 bytes. 2026/07/16 13:58:36 13:58:01 INFO Pod/kserve-controller-manager-6c67dcb69c-nskhp: AddedInterface - Add eth0 [10.130.1.173/23] from ovn-kubernetes 2026/07/16 13:58:36 13:58:01 INFO Pod/kserve-controller-manager-6c67dcb69c-nskhp: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/16 13:58:36 13:58:01 INFO Pod/kserve-controller-manager-6c67dcb69c-nskhp: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 688ms (688ms including waiting). Image size: 238435613 bytes. 2026/07/16 13:58:36 13:58:01 INFO Pod/kserve-controller-manager-6c67dcb69c-nskhp: Created - Created container: manager 2026/07/16 13:58:36 13:58:01 INFO Pod/llmisvc-controller-manager-5d65df97bb-7ftqw: AddedInterface - Add eth0 [10.130.1.174/23] from ovn-kubernetes 2026/07/16 13:58:36 13:58:01 INFO Pod/llmisvc-controller-manager-5d65df97bb-7ftqw: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/16 13:58:36 13:58:00 INFO Pod/llmisvc-controller-manager-5d65df97bb-7ftqw: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-7ftqw to ip-10-0-1-189.us-east-2.compute.internal 2026/07/16 13:58:36 13:58:00 INFO Pod/kserve-controller-manager-6c67dcb69c-nskhp: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-nskhp to ip-10-0-1-189.us-east-2.compute.internal 2026/07/16 13:58:36 13:58:00 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-7ftqw 2026/07/16 13:58:36 13:58:00 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/16 13:58:36 13:58:00 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-nskhp 2026/07/16 13:58:36 13:58:00 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/16 13:58:36 13:57:50 INFO Pod/model-serving-api-6688cd56bd-pprhd: Created - Created container: server 2026/07/16 13:58:36 13:57:50 INFO Pod/model-serving-api-6688cd56bd-pprhd: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/16 13:58:36 13:57:50 INFO Pod/model-serving-api-6688cd56bd-pprhd: AddedInterface - Add eth0 [10.130.1.172/23] from ovn-kubernetes 2026/07/16 13:58:36 13:57:50 INFO Pod/model-serving-api-6688cd56bd-pprhd: Started - Started container server 2026/07/16 13:58:36 13:57:49 INFO Pod/model-serving-api-6688cd56bd-pprhd: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-pprhd to ip-10-0-1-189.us-east-2.compute.internal 2026/07/16 13:58:36 13:57:49 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 13:58:36 13:57:49 WARN Pod/model-serving-api-6688cd56bd-pprhd: FailedMount - MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 2026/07/16 13:58:36 13:57:49 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-pprhd 2026/07/16 13:58:36 13:57:48 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/16 13:58:36 13:57:48 INFO Pod/odh-model-controller-6849fc49d8-f2q2t: Started - Started container manager 2026/07/16 13:58:36 13:57:48 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/16 13:58:36 13:57:48 INFO Pod/odh-model-controller-6849fc49d8-f2q2t: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 551ms (551ms including waiting). Image size: 238959091 bytes. 2026/07/16 13:58:36 13:57:48 INFO Pod/odh-model-controller-6849fc49d8-f2q2t: Created - Created container: manager 2026/07/16 13:58:36 13:57:47 INFO Pod/odh-model-controller-6849fc49d8-f2q2t: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/16 13:58:36 13:57:47 INFO Pod/odh-model-controller-6849fc49d8-f2q2t: AddedInterface - Add eth0 [10.130.1.171/23] from ovn-kubernetes 2026/07/16 13:58:36 13:57:46 INFO Pod/odh-model-controller-6849fc49d8-f2q2t: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-f2q2t to ip-10-0-1-189.us-east-2.compute.internal 2026/07/16 13:58:36 13:57:46 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/16 13:58:36 13:57:46 WARN Pod/odh-model-controller-6849fc49d8-f2q2t: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/16 13:58:36 13:57:46 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-f2q2t 2026/07/16 13:58:36 13:57:36 INFO Pod/kserve-controller-manager-6c67dcb69c-27vv8: Killing - Stopping container manager 2026/07/16 13:58:36 13:57:36 INFO Pod/model-serving-api-6688cd56bd-x9q6v: Killing - Stopping container server 2026/07/16 13:58:36 13:57:36 INFO Pod/llmisvc-controller-manager-5d65df97bb-6phrd: Killing - Stopping container manager 2026/07/16 13:58:36 13:57:36 INFO Pod/odh-model-controller-6849fc49d8-kssf5: Killing - Stopping container manager 2026/07/16 13:58:36 13:57:36 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 13:58:36 13:57:13 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-27vv8_a401d405-01af-47ec-8f02-51510db3959c became leader 2026/07/16 13:58:36 13:57:13 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-6phrd_36e73fc6-8091-4a00-b75d-2d0d26265da9 became leader 2026/07/16 13:58:36 13:56:57 INFO Pod/kserve-controller-manager-6c67dcb69c-27vv8: Started - Started container manager 2026/07/16 13:58:36 13:56:57 INFO Pod/llmisvc-controller-manager-5d65df97bb-6phrd: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 584ms (584ms including waiting). Image size: 236366599 bytes. 2026/07/16 13:58:36 13:56:57 INFO Pod/llmisvc-controller-manager-5d65df97bb-6phrd: Created - Created container: manager 2026/07/16 13:58:36 13:56:57 INFO Pod/kserve-controller-manager-6c67dcb69c-27vv8: Created - Created container: manager 2026/07/16 13:58:36 13:56:57 INFO Pod/kserve-controller-manager-6c67dcb69c-27vv8: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 657ms (657ms including waiting). Image size: 238435613 bytes. 2026/07/16 13:58:36 13:56:57 INFO Pod/llmisvc-controller-manager-5d65df97bb-6phrd: Started - Started container manager 2026/07/16 13:58:36 13:56:56 INFO Pod/llmisvc-controller-manager-5d65df97bb-6phrd: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-6phrd to ip-10-0-1-189.us-east-2.compute.internal 2026/07/16 13:58:36 13:56:56 INFO Pod/kserve-controller-manager-6c67dcb69c-27vv8: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-27vv8 to ip-10-0-1-189.us-east-2.compute.internal 2026/07/16 13:58:36 13:56:56 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/16 13:58:36 13:56:56 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/16 13:58:36 13:56:56 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-27vv8 2026/07/16 13:58:36 13:56:56 INFO Pod/kserve-controller-manager-6c67dcb69c-27vv8: AddedInterface - Add eth0 [10.130.1.169/23] from ovn-kubernetes 2026/07/16 13:58:36 13:56:56 INFO Pod/llmisvc-controller-manager-5d65df97bb-6phrd: AddedInterface - Add eth0 [10.130.1.170/23] from ovn-kubernetes 2026/07/16 13:58:36 13:56:56 INFO Pod/llmisvc-controller-manager-5d65df97bb-6phrd: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/16 13:58:36 13:56:56 INFO Pod/kserve-controller-manager-6c67dcb69c-27vv8: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/16 13:58:36 13:56:56 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-6phrd 2026/07/16 13:58:36 13:56:55 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-kssf5_3c47b7f0-c63d-4106-84e7-32d7cd18687e became leader 2026/07/16 13:58:36 13:56:46 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-7ccfdd66-97jp9_b1a93684-3b95-4b4f-a148-5a55ab627331 became leader 2026/07/16 13:58:36 13:56:37 INFO Pod/model-serving-api-6688cd56bd-x9q6v: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-x9q6v to ip-10-0-1-189.us-east-2.compute.internal 2026/07/16 13:58:36 13:56:37 INFO Pod/model-serving-api-6688cd56bd-x9q6v: Started - Started container server 2026/07/16 13:58:36 13:56:37 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/16 13:58:36 13:56:37 INFO Pod/model-serving-api-6688cd56bd-x9q6v: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/16 13:58:36 13:56:37 INFO Pod/model-serving-api-6688cd56bd-x9q6v: Created - Created container: server 2026/07/16 13:58:36 13:56:37 INFO Pod/model-serving-api-6688cd56bd-x9q6v: AddedInterface - Add eth0 [10.130.1.168/23] from ovn-kubernetes 2026/07/16 13:58:36 13:56:37 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-x9q6v 2026/07/16 13:58:36 13:56:36 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/16 13:58:36 13:56:36 INFO Pod/odh-model-controller-6849fc49d8-kssf5: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/16 13:58:36 13:56:36 INFO Pod/odh-model-controller-6849fc49d8-kssf5: AddedInterface - Add eth0 [10.130.1.167/23] from ovn-kubernetes 2026/07/16 13:58:36 13:56:36 INFO Pod/odh-model-controller-6849fc49d8-kssf5: Started - Started container manager 2026/07/16 13:58:36 13:56:36 INFO Pod/odh-model-controller-6849fc49d8-kssf5: Created - Created container: manager 2026/07/16 13:58:36 13:56:36 INFO Pod/odh-model-controller-6849fc49d8-kssf5: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 682ms (682ms including waiting). Image size: 238959091 bytes. 2026/07/16 13:58:36 13:56:35 WARN Pod/odh-model-controller-6849fc49d8-kssf5: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/16 13:58:36 13:56:34 INFO Pod/odh-model-controller-6849fc49d8-kssf5: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-kssf5 to ip-10-0-1-189.us-east-2.compute.internal 2026/07/16 13:58:36 13:56:34 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/16 13:58:36 13:56:34 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-kssf5 2026/07/16 13:58:36 13:56:29 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-97jp9: Created - Created container: manager 2026/07/16 13:58:36 13:56:29 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-97jp9: Started - Started container manager 2026/07/16 13:58:36 13:56:29 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-97jp9: Pulled - Container image "quay.io/rhoai/odh-trainer-rhel9@sha256:59ea523e2df9c1e705c755308ac11420682081cc63d54890e2b15fbd7d8ba570" already present on machine 2026/07/16 13:58:36 13:56:29 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-97jp9: AddedInterface - Add eth0 [10.130.1.166/23] from ovn-kubernetes 2026/07/16 13:58:36 13:56:28 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-97jp9: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-7ccfdd66-97jp9 to ip-10-0-1-189.us-east-2.compute.internal 2026/07/16 13:58:36 13:56:28 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-7ccfdd66 from 0 to 1 2026/07/16 13:58:36 13:56:28 INFO ReplicaSet/kubeflow-trainer-controller-manager-7ccfdd66: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-7ccfdd66-97jp9 2026/07/16 13:58:36 13:56:25 INFO Pod/model-serving-api-6688cd56bd-ffbhk: Killing - Stopping container server 2026/07/16 13:58:36 13:56:25 INFO Pod/kserve-controller-manager-6c67dcb69c-qnvtf: Killing - Stopping container manager 2026/07/16 13:58:36 13:56:25 INFO Pod/odh-model-controller-6849fc49d8-9t4mx: Killing - Stopping container manager 2026/07/16 13:58:36 13:56:25 INFO Pod/llmisvc-controller-manager-5d65df97bb-mdhvr: Killing - Stopping container manager 2026/07/16 13:58:36 13:55:13 INFO Pod/maas-controller-85bdcf9988-d6l59: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 647ms (647ms including waiting). Image size: 212866077 bytes. 2026/07/16 13:58:36 13:55:12 INFO Pod/maas-controller-85bdcf9988-d6l59: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/16 13:58:36 13:54:17 WARN Pod/maas-controller-85bdcf9988-d6l59: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-94f49); attempting to pull the image may not succeed. 2026/07/16 13:58:36 === RESOURCE QUOTAS === 2026/07/16 13:58:36 No resource quotas found in monitored namespaces 2026/07/16 13:58:36 Classification for components: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/16 13:58:36 Evidence: container maas-controller-85bdcf9988-d6l59/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-d6l59_redhat-ods-applications(b5835c58-4c0c-4fbf-ad57-98396b8258ee) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-85bdcf9988-d6l59/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-d6l59_redhat-ods-applications(b5835c58-4c0c-4fbf-ad57-98396b8258ee)"],"confidence":"medium"} 2026/07/16 13:58:36 Diagnostics complete for failed test: components controller_test.go:484: Stopping: components test failed. 2026/07/16 13:58:36 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/16 13:58:36 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/16 13:58:37 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (1498.96s)