=== RUN TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable helper_test.go:150: Verifying Tenant models-as-a-service/default-tenant is present before disable helper_test.go:150: Disabling MaaS subcomponent (setting to Removed) helper_test.go:150: Waiting for Tenant models-as-a-service/default-tenant to be deleted testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable (430.21s) === RUN TestOdhOperator/components/group_4/modelsasservice === PAUSE TestOdhOperator/components/group_4/modelsasservice === CONT TestOdhOperator/components/group_4/modelsasservice 2026-07-09T22:31:57Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.20.12"}} controller_test.go:753: Creating MaaS PostgreSQL instance in namespace redhat-ods-applications controller_test.go:753: MaaS PostgreSQL instance and maas-db-config secret created in namespace redhat-ods-applications controller_test.go:753: Creating MaaS GatewayClass and Gateway: openshift-ingress/maas-default-gateway controller_test.go:753: Using hostname=maas.apps.rosa.rh-nightly-pm.nuth.p3.openshiftapps.com, certSecret=2o8clk8b6cj832lmkd2oj78crnf7qn6d-primary-cert-bundle-secret controller_test.go:753: MaaS Gateway openshift-ingress/maas-default-gateway created with HTTP+HTTPS listeners test_context_test.go:615: Timed out after 420.002s. The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1824 with: Expected resource list 'models-as-a-service/default-tenant' of kind 'Tenant' to be empty, but it contains resources. Expected <[]unstructured.Unstructured | len:1, cap:1>: [ { Object: { "metadata": <map[string]interface {} | len:8>{ "generation": <int64>1, "managedFields": <[]interface {} | len:2, cap:2>[ <map[string]interface {} | len:6>{ "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:2>{ "f:metadata": <map[string]interface {} | len:1>{ "f:ownerReferences": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "k:{\"uid\":\"e5f1d992-67dc-4518-b198-b32c7565359d\"}": <map[string]interface {} | len:0>{}, }, }, "f:spec": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "f:gatewayRef": <map[string]interface {} | len:3>{ "f:name": <map[string]interface {} | len:0>{}, "f:namespace": <map[string]interface {} | len:0>{}, ".": <map[string]interface {} | len:0>{}, }, }, }, "manager": <string>"manager", "operation": <string>"Update", "time": <string>"2026-07-09T22:33:05Z", }, <map[string]interface {} | len:7>{ "manager": <string>"manager", "operation": <string>"Update", "subresource": <string>"status", "time": <string>"2026-07-09T22:33:29Z", "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:1>{ "f:status": <map[string]interface {} | len:3>{ "f:phase": <map[string]interface {} | len:0>{}, ".": <map[string]interface {} | len:0>{}, "f:conditions": <map[string]interface {} | len:0>{}, }, }, }, ], "name": <string>"default-tenant", "namespace": <string>"models-as-a-service", "ownerReferences": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:4>{ "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "kind": <string>"Config", "name": <string>"default", "uid": <string>"e5f1d992-67dc-4518-b198-b32c7565359d", }, ], "resourceVersion": <string>"208341166", "uid": <string>"055acee8-6579-436a-a89e-0d9599466f7c", "creationTimestamp": <string>"2026-07-09T22:33:05Z", }, "spec": <map[string]interface {} | len:1>{ "gatewayRef": <map[string]interface {} | len:2>{ "name": <string>"maas-default-gateway", "namespace": <string>"openshift-ingress", }, }, "status": <map[string]interface {} | len:2>{ "phase": <string>"Active", "conditions": <[]interface {} | len:5, cap:8>[ <map[string]interface {} | len:6>{ "observedGeneration": <int64>1, "reason": <string>"DependenciesMet", "status": <string>"True", "type": <string>"DependenciesAvailable", "lastTransitionTime": <string>"2026-07-09T22:33:08Z", "message": <string>"AuthConfig CRD (Kuadrant) is available on the cluster", }, <map[string]interface {} | len:6>{ "type": <string>"MaaSPrerequisitesAvailable", "lastTransitionTime": <string>"2026-07-09T22:33:08Z", "message": <string>"All prerequisites are satisfied", "observedGeneration": <int64>1, "reason": <string>"PrerequisitesMet", "status": <string>"True", }, <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-07-09T22:33:08Z", "message": <string>"All prerequisites are satisfied", "observedGeneration": <int64>1, "reason": <string>"PrerequisitesMet", "status": <string>"False", "type": <string>"Degraded", }, <map[string]interface {} | len:6>{ "status": <string>"True", "type": <string>"DeploymentsAvailable", "lastTransitionTime": <string>"2026-07-09T22:33:29Z", "message": <string>"maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"DeploymentsReady", }, <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-07-09T22:33:29Z", "message": <string>"MaaS platform manifests applied and maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"Reconciled", "status": <string>"True", "type": <string>"Ready", }, ], }, "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "kind": <string>"Tenant", }, }, ] to be empty --- FAIL: TestOdhOperator/components/group_4/modelsasservice (531.77s) === RUN TestOdhOperator/components/group_4 --- FAIL: TestOdhOperator/components/group_4 (0.00s) === RUN TestOdhOperator/components 2026/07/09 22:40:48 TEST FAILURE DETECTED: group 4 2026/07/09 22:40:48 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 4) === 2026/07/09 22:40:49 === CLUSTER STATE === 2026/07/09 22:40:49 Node: ip-10-0-1-10.us-east-2.compute.internal 2026/07/09 22:40:49 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 22:40:49 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 22:40:49 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 22:40:49 Condition Ready: True - kubelet is posting ready status 2026/07/09 22:40:49 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/09 22:40:49 Node: ip-10-0-1-132.us-east-2.compute.internal 2026/07/09 22:40:49 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 22:40:49 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 22:40:49 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 22:40:49 Condition Ready: True - kubelet is posting ready status 2026/07/09 22:40:49 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/09 22:40:49 Node: ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 22:40:49 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 22:40:49 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 22:40:49 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 22:40:49 Condition Ready: True - kubelet is posting ready status 2026/07/09 22:40:49 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/09 22:40:49 === DEPLOYMENTS === 2026/07/09 22:40:49 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/09 22:40:49 === PODS === 2026/07/09 22:40:49 Failed to collect pod data: redhat-ods-applications/maas-controller-85bdcf9988-89hlk: container manager not ready 2026/07/09 22:40:49 === OPERATOR STATUS === 2026/07/09 22:40:49 Operator deployment: 3/3 ready 2026/07/09 22:40:49 === DSCI STATUS === 2026/07/09 22:40:49 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/09 22:40:49 === DSC STATUS === 2026/07/09 22:40:49 DSC default-dsc: 2026/07/09 22:40:49 Ready: True - 2026/07/09 22:40:49 ProvisioningSucceeded: True - 2026/07/09 22:40:49 ComponentsReady: True - 2026/07/09 22:40:49 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/09 22:40:49 DashboardReady: False - Component ManagementState is set to Removed 2026/07/09 22:40:49 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 22:40:49 KserveLLMInferenceServiceDependencies: True - 2026/07/09 22:40:49 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/09 22:40:49 KserveReady: True - 2026/07/09 22:40:49 KueueReady: False - Component ManagementState is set to Removed 2026/07/09 22:40:49 LLM-D-WVADependencies: True - 2026/07/09 22:40:49 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 22:40:49 ModelControllerReady: True - 2026/07/09 22:40:49 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/09 22:40:49 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/09 22:40:49 OGXReady: False - Component ManagementState is set to Removed 2026/07/09 22:40:49 RayReady: False - Component ManagementState is set to Removed 2026/07/09 22:40:49 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 22:40:49 TrainerReady: False - Component ManagementState is set to Removed 2026/07/09 22:40:49 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 22:40:49 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/09 22:40:49 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/09 22:40:49 === RECENT EVENTS === 2026/07/09 22:40:49 22:39:46 INFO Pod/maas-controller-85bdcf9988-89hlk: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/09 22:40:49 22:39:46 INFO Pod/maas-controller-85bdcf9988-89hlk: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 518ms (518ms including waiting). Image size: 212866077 bytes. 2026/07/09 22:40:49 22:38:53 WARN Pod/maas-controller-85bdcf9988-89hlk: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-d7w6h); attempting to pull the image may not succeed. 2026/07/09 22:40:49 22:36:59 INFO Pod/maas-controller-85bdcf9988-89hlk: Created - Created container: manager 2026/07/09 22:40:49 22:36:59 INFO Pod/maas-controller-85bdcf9988-89hlk: Started - Started container manager 2026/07/09 22:40:49 22:36:59 INFO Pod/maas-controller-85bdcf9988-89hlk: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 538ms (538ms including waiting). Image size: 212866077 bytes. 2026/07/09 22:40:49 === RESOURCE QUOTAS === 2026/07/09 22:40:49 No resource quotas found in monitored namespaces 2026/07/09 22:40:49 Classification for group 4: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/09 22:40:49 Evidence: container maas-controller-85bdcf9988-89hlk/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-89hlk_redhat-ods-applications(a23de847-bfb6-4b1e-b778-0b2f09726c70) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-85bdcf9988-89hlk/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-89hlk_redhat-ods-applications(a23de847-bfb6-4b1e-b778-0b2f09726c70)"],"confidence":"medium"} 2026/07/09 22:40:49 Diagnostics complete for failed test: group 4 controller_test.go:753: Stopping: group 4 test failed. 2026/07/09 22:40:49 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/09 22:40:49 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/09 22:42:00 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (1444.93s) === RUN TestOdhOperator 2026/07/09 22:18:45 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/09 22:43:01 TEST FAILURE DETECTED: components 2026/07/09 22:43:01 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/09 22:43:01 === CLUSTER STATE === 2026/07/09 22:43:01 Node: ip-10-0-1-10.us-east-2.compute.internal 2026/07/09 22:43:01 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 22:43:01 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 22:43:01 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 22:43:01 Condition Ready: True - kubelet is posting ready status 2026/07/09 22:43:01 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/09 22:43:01 Node: ip-10-0-1-132.us-east-2.compute.internal 2026/07/09 22:43:01 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 22:43:01 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 22:43:01 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 22:43:01 Condition Ready: True - kubelet is posting ready status 2026/07/09 22:43:01 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/09 22:43:01 Node: ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 22:43:01 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 22:43:01 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 22:43:01 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 22:43:01 Condition Ready: True - kubelet is posting ready status 2026/07/09 22:43:01 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/09 22:43:01 === DEPLOYMENTS === 2026/07/09 22:43:01 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/09 22:43:01 === PODS === 2026/07/09 22:43:01 Failed to collect pod data: redhat-ods-applications/maas-controller-85bdcf9988-89hlk: container manager not ready 2026/07/09 22:43:01 === OPERATOR STATUS === 2026/07/09 22:43:01 Operator deployment: 3/3 ready 2026/07/09 22:43:01 === DSCI STATUS === 2026/07/09 22:43:01 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/09 22:43:01 === DSC STATUS === 2026/07/09 22:43:01 DSC default-dsc: 2026/07/09 22:43:01 Ready: True - 2026/07/09 22:43:01 ProvisioningSucceeded: True - 2026/07/09 22:43:01 ComponentsReady: True - 2026/07/09 22:43:01 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/09 22:43:01 DashboardReady: False - Component ManagementState is set to Removed 2026/07/09 22:43:01 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 22:43:01 KserveLLMInferenceServiceDependencies: True - 2026/07/09 22:43:01 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/09 22:43:01 KserveReady: True - 2026/07/09 22:43:01 KueueReady: False - Component ManagementState is set to Removed 2026/07/09 22:43:01 LLM-D-WVADependencies: True - 2026/07/09 22:43:01 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 22:43:01 ModelControllerReady: True - 2026/07/09 22:43:01 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/09 22:43:01 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/09 22:43:01 OGXReady: False - Component ManagementState is set to Removed 2026/07/09 22:43:01 RayReady: False - Component ManagementState is set to Removed 2026/07/09 22:43:01 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 22:43:01 TrainerReady: True - 2026/07/09 22:43:01 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 22:43:01 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/09 22:43:01 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/09 22:43:01 === RECENT EVENTS === 2026/07/09 22:43:01 22:42:45 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-hqtxn_02cf7e07-024f-4862-a5bf-92403ca844e3 became leader 2026/07/09 22:43:01 22:42:42 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-5blln_8fa5cbce-c3a2-4a7f-af18-239c0eb3bb71 became leader 2026/07/09 22:43:01 22:42:29 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-bnhtb_4d9e742a-b75d-451e-818e-5e24d3612415 became leader 2026/07/09 22:43:01 22:42:26 INFO Pod/llmisvc-controller-manager-5d65df97bb-hqtxn: Started - Started container manager 2026/07/09 22:43:01 22:42:26 INFO Pod/kserve-controller-manager-6c67dcb69c-5blln: Created - Created container: manager 2026/07/09 22:43:01 22:42:26 INFO Pod/kserve-controller-manager-6c67dcb69c-5blln: Started - Started container manager 2026/07/09 22:43:01 22:42:26 INFO Pod/kserve-controller-manager-6c67dcb69c-5blln: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/09 22:43:01 22:42:26 INFO Pod/llmisvc-controller-manager-5d65df97bb-hqtxn: AddedInterface - Add eth0 [10.128.1.211/23] from ovn-kubernetes 2026/07/09 22:43:01 22:42:26 INFO Pod/llmisvc-controller-manager-5d65df97bb-hqtxn: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/09 22:43:01 22:42:26 INFO Pod/llmisvc-controller-manager-5d65df97bb-hqtxn: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 522ms (522ms including waiting). Image size: 236366599 bytes. 2026/07/09 22:43:01 22:42:26 INFO Pod/kserve-controller-manager-6c67dcb69c-5blln: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 480ms (480ms including waiting). Image size: 238435613 bytes. 2026/07/09 22:43:01 22:42:26 INFO Pod/kserve-controller-manager-6c67dcb69c-5blln: AddedInterface - Add eth0 [10.130.11.174/23] from ovn-kubernetes 2026/07/09 22:43:01 22:42:26 INFO Pod/llmisvc-controller-manager-5d65df97bb-hqtxn: Created - Created container: manager 2026/07/09 22:43:01 22:42:25 INFO Pod/llmisvc-controller-manager-5d65df97bb-hqtxn: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-hqtxn to ip-10-0-1-10.us-east-2.compute.internal 2026/07/09 22:43:01 22:42:25 INFO Pod/kserve-controller-manager-6c67dcb69c-5blln: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-5blln to ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 22:43:01 22:42:25 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-hqtxn 2026/07/09 22:43:01 22:42:25 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-5blln 2026/07/09 22:43:01 22:42:25 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/09 22:43:01 22:42:25 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/09 22:43:01 22:42:13 INFO Pod/model-serving-api-6688cd56bd-xck4q: Started - Started container server 2026/07/09 22:43:01 22:42:13 INFO Pod/model-serving-api-6688cd56bd-xck4q: Created - Created container: server 2026/07/09 22:43:01 22:42:13 INFO Pod/model-serving-api-6688cd56bd-xck4q: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/09 22:43:01 22:42:13 INFO Pod/model-serving-api-6688cd56bd-xck4q: AddedInterface - Add eth0 [10.130.11.173/23] from ovn-kubernetes 2026/07/09 22:43:01 22:42:12 INFO Pod/model-serving-api-6688cd56bd-xck4q: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-xck4q to ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 22:43:01 22:42:12 INFO Pod/odh-model-controller-6849fc49d8-bnhtb: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 409ms (409ms including waiting). Image size: 238959091 bytes. 2026/07/09 22:43:01 22:42:12 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/09 22:43:01 22:42:12 INFO Pod/odh-model-controller-6849fc49d8-bnhtb: Created - Created container: manager 2026/07/09 22:43:01 22:42:12 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/09 22:43:01 22:42:12 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-xck4q 2026/07/09 22:43:01 22:42:12 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/09 22:43:01 22:42:12 INFO Pod/odh-model-controller-6849fc49d8-bnhtb: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/09 22:43:01 22:42:12 INFO Pod/odh-model-controller-6849fc49d8-bnhtb: AddedInterface - Add eth0 [10.128.1.210/23] from ovn-kubernetes 2026/07/09 22:43:01 22:42:12 INFO Pod/odh-model-controller-6849fc49d8-bnhtb: Started - Started container manager 2026/07/09 22:43:01 22:42:11 INFO Pod/odh-model-controller-6849fc49d8-bnhtb: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-bnhtb to ip-10-0-1-10.us-east-2.compute.internal 2026/07/09 22:43:01 22:42:11 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/09 22:43:01 22:42:11 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-bnhtb 2026/07/09 22:43:01 22:42:11 WARN Pod/odh-model-controller-6849fc49d8-bnhtb: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/09 22:43:01 22:42:01 INFO Pod/odh-model-controller-6849fc49d8-htglh: Killing - Stopping container manager 2026/07/09 22:43:01 22:42:01 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/09 22:43:01 22:42:01 INFO Pod/model-serving-api-6688cd56bd-qspp6: Killing - Stopping container server 2026/07/09 22:43:01 22:42:01 WARN Pod/kserve-controller-manager-6c67dcb69c-kvmg2: Unhealthy - Readiness probe failed: Get "http://10.130.11.172:8081/readyz": dial tcp 10.130.11.172:8081: connect: connection refused 2026/07/09 22:43:01 22:42:01 INFO Pod/kserve-controller-manager-6c67dcb69c-kvmg2: Killing - Stopping container manager 2026/07/09 22:43:01 22:42:01 INFO Pod/llmisvc-controller-manager-5d65df97bb-hjprq: Killing - Stopping container manager 2026/07/09 22:43:01 22:41:36 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-kvmg2_c264a30d-0fcc-43c0-a4c4-863c6a46be63 became leader 2026/07/09 22:43:01 22:41:36 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-hjprq_7cf065b7-015d-4c26-af55-c1edc2d570b1 became leader 2026/07/09 22:43:01 22:41:21 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-htglh_8b23716b-ddc2-4f5a-ba8c-54ee74e09e81 became leader 2026/07/09 22:43:01 22:41:20 INFO Pod/llmisvc-controller-manager-5d65df97bb-hjprq: Created - Created container: manager 2026/07/09 22:43:01 22:41:20 INFO Pod/llmisvc-controller-manager-5d65df97bb-hjprq: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 643ms (643ms including waiting). Image size: 236366599 bytes. 2026/07/09 22:43:01 22:41:20 INFO Pod/llmisvc-controller-manager-5d65df97bb-hjprq: Started - Started container manager 2026/07/09 22:43:01 22:41:19 INFO Pod/kserve-controller-manager-6c67dcb69c-kvmg2: Started - Started container manager 2026/07/09 22:43:01 22:41:19 INFO Pod/llmisvc-controller-manager-5d65df97bb-hjprq: AddedInterface - Add eth0 [10.128.1.209/23] from ovn-kubernetes 2026/07/09 22:43:01 22:41:19 INFO Pod/llmisvc-controller-manager-5d65df97bb-hjprq: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/09 22:43:01 22:41:19 INFO Pod/kserve-controller-manager-6c67dcb69c-kvmg2: Created - Created container: manager 2026/07/09 22:43:01 22:41:19 INFO Pod/kserve-controller-manager-6c67dcb69c-kvmg2: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 620ms (620ms including waiting). Image size: 238435613 bytes. 2026/07/09 22:43:01 22:41:19 INFO Pod/kserve-controller-manager-6c67dcb69c-kvmg2: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/09 22:43:01 22:41:19 INFO Pod/kserve-controller-manager-6c67dcb69c-kvmg2: AddedInterface - Add eth0 [10.130.11.172/23] from ovn-kubernetes 2026/07/09 22:43:01 22:41:18 INFO Pod/llmisvc-controller-manager-5d65df97bb-hjprq: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-hjprq to ip-10-0-1-10.us-east-2.compute.internal 2026/07/09 22:43:01 22:41:18 INFO Pod/kserve-controller-manager-6c67dcb69c-kvmg2: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-kvmg2 to ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 22:43:01 22:41:18 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-hjprq 2026/07/09 22:43:01 22:41:18 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/09 22:43:01 22:41:18 WARN Pod/llmisvc-controller-manager-5d65df97bb-hjprq: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 2026/07/09 22:43:01 22:41:18 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/09 22:43:01 22:41:18 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-kvmg2 2026/07/09 22:43:01 22:41:10 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-7ccfdd66-d6phz_94e797a5-a451-4f6d-9954-e26f773e3f3b became leader 2026/07/09 22:43:01 22:41:03 INFO Pod/model-serving-api-6688cd56bd-qspp6: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/09 22:43:01 22:41:03 INFO Pod/model-serving-api-6688cd56bd-qspp6: AddedInterface - Add eth0 [10.128.1.208/23] from ovn-kubernetes 2026/07/09 22:43:01 22:41:03 INFO Pod/odh-model-controller-6849fc49d8-htglh: Started - Started container manager 2026/07/09 22:43:01 22:41:03 INFO Pod/model-serving-api-6688cd56bd-qspp6: Started - Started container server 2026/07/09 22:43:01 22:41:03 INFO Pod/odh-model-controller-6849fc49d8-htglh: Created - Created container: manager 2026/07/09 22:43:01 22:41:03 INFO Pod/model-serving-api-6688cd56bd-qspp6: Created - Created container: server 2026/07/09 22:43:01 22:41:03 INFO Pod/odh-model-controller-6849fc49d8-htglh: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 969ms (969ms including waiting). Image size: 238959091 bytes. 2026/07/09 22:43:01 22:41:02 INFO Pod/model-serving-api-6688cd56bd-qspp6: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-qspp6 to ip-10-0-1-10.us-east-2.compute.internal 2026/07/09 22:43:01 22:41:02 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-qspp6 2026/07/09 22:43:01 22:41:02 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/09 22:43:01 22:41:02 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/09 22:43:01 22:41:02 INFO Pod/odh-model-controller-6849fc49d8-htglh: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/09 22:43:01 22:41:02 INFO Pod/odh-model-controller-6849fc49d8-htglh: AddedInterface - Add eth0 [10.130.11.171/23] from ovn-kubernetes 2026/07/09 22:43:01 22:41:01 INFO Pod/odh-model-controller-6849fc49d8-htglh: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-htglh to ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 22:43:01 22:41:01 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-htglh 2026/07/09 22:43:01 22:41:01 WARN Pod/odh-model-controller-6849fc49d8-htglh: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/09 22:43:01 22:41:00 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/09 22:43:01 22:40:54 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-d6phz: Pulled - Container image "quay.io/rhoai/odh-trainer-rhel9@sha256:59ea523e2df9c1e705c755308ac11420682081cc63d54890e2b15fbd7d8ba570" already present on machine 2026/07/09 22:43:01 22:40:54 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-d6phz: Created - Created container: manager 2026/07/09 22:43:01 22:40:54 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-d6phz: Started - Started container manager 2026/07/09 22:43:01 22:40:53 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-d6phz: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-7ccfdd66-d6phz to ip-10-0-1-10.us-east-2.compute.internal 2026/07/09 22:43:01 22:40:53 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-7ccfdd66 from 0 to 1 2026/07/09 22:43:01 22:40:53 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-d6phz: AddedInterface - Add eth0 [10.128.1.207/23] from ovn-kubernetes 2026/07/09 22:43:01 22:40:53 INFO ReplicaSet/kubeflow-trainer-controller-manager-7ccfdd66: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-7ccfdd66-d6phz 2026/07/09 22:43:01 22:40:50 INFO Pod/kserve-controller-manager-6c67dcb69c-gxzk4: Killing - Stopping container manager 2026/07/09 22:43:01 22:40:50 INFO Pod/llmisvc-controller-manager-5d65df97bb-ngfxc: Killing - Stopping container manager 2026/07/09 22:43:01 22:40:50 INFO Pod/odh-model-controller-6849fc49d8-jpsh7: Killing - Stopping container manager 2026/07/09 22:43:01 22:40:50 INFO Pod/model-serving-api-6688cd56bd-jlznz: Killing - Stopping container server 2026/07/09 22:43:01 22:39:46 INFO Pod/maas-controller-85bdcf9988-89hlk: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 518ms (518ms including waiting). Image size: 212866077 bytes. 2026/07/09 22:43:01 22:39:46 INFO Pod/maas-controller-85bdcf9988-89hlk: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/09 22:43:01 22:38:53 WARN Pod/maas-controller-85bdcf9988-89hlk: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-d7w6h); attempting to pull the image may not succeed. 2026/07/09 22:43:01 === RESOURCE QUOTAS === 2026/07/09 22:43:01 No resource quotas found in monitored namespaces 2026/07/09 22:43:01 Classification for components: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/09 22:43:01 Evidence: container maas-controller-85bdcf9988-89hlk/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-89hlk_redhat-ods-applications(a23de847-bfb6-4b1e-b778-0b2f09726c70) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-85bdcf9988-89hlk/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-89hlk_redhat-ods-applications(a23de847-bfb6-4b1e-b778-0b2f09726c70)"],"confidence":"medium"} 2026/07/09 22:43:01 Diagnostics complete for failed test: components controller_test.go:484: Stopping: components test failed. 2026/07/09 22:43:01 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/09 22:43:01 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/09 22:43:02 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (1457.59s)