=== RUN TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable helper_test.go:150: Verifying Tenant models-as-a-service/default-tenant is present before disable helper_test.go:150: Disabling MaaS subcomponent (setting to Removed) helper_test.go:150: Waiting for Tenant models-as-a-service/default-tenant to be deleted testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable (430.53s) === RUN TestOdhOperator/components/group_4/modelsasservice === PAUSE TestOdhOperator/components/group_4/modelsasservice === CONT TestOdhOperator/components/group_4/modelsasservice 2026-07-15T12:33:03Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.20.14"}} controller_test.go:753: Creating MaaS PostgreSQL instance in namespace redhat-ods-applications controller_test.go:753: MaaS PostgreSQL instance and maas-db-config secret created in namespace redhat-ods-applications controller_test.go:753: Creating MaaS GatewayClass and Gateway: openshift-ingress/maas-default-gateway controller_test.go:753: Using hostname=maas.apps.ods-qe-psi-23.osp.rh-ods.com, certSecret=router-certs-default controller_test.go:753: MaaS Gateway openshift-ingress/maas-default-gateway created with HTTP+HTTPS listeners test_context_test.go:615: Timed out after 420.001s. The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1824 with: Expected resource list 'models-as-a-service/default-tenant' of kind 'Tenant' to be empty, but it contains resources. Expected <[]unstructured.Unstructured | len:1, cap:1>: [ { Object: { "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "kind": <string>"Tenant", "metadata": <map[string]interface {} | len:8>{ "name": <string>"default-tenant", "namespace": <string>"models-as-a-service", "ownerReferences": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:4>{ "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "kind": <string>"Config", "name": <string>"default", "uid": <string>"72e913fb-d332-4772-8485-88b5e5534fdb", }, ], "resourceVersion": <string>"272811283", "uid": <string>"8ff456c7-f251-4fa3-88fb-2a29d9714215", "creationTimestamp": <string>"2026-07-15T12:33:51Z", "generation": <int64>1, "managedFields": <[]interface {} | len:2, cap:2>[ <map[string]interface {} | len:6>{ "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:2>{ "f:metadata": <map[string]interface {} | len:1>{ "f:ownerReferences": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "k:{\"uid\":\"72e913fb-d332-4772-8485-88b5e5534fdb\"}": <map[string]interface {} | len:0>{}, }, }, "f:spec": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "f:gatewayRef": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:name": <map[string]interface {} | len:0>{}, "f:namespace": <map[string]interface {} | len:0>{}, }, }, }, "manager": <string>"manager", "operation": <string>"Update", "time": <string>"2026-07-15T12:33:51Z", }, <map[string]interface {} | len:7>{ "fieldsV1": <map[string]interface {} | len:1>{ "f:status": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:conditions": <map[string]interface {} | len:0>{}, "f:phase": <map[string]interface {} | len:0>{}, }, }, "manager": <string>"manager", "operation": <string>"Update", "subresource": <string>"status", "time": <string>"2026-07-15T12:34:30Z", "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "fieldsType": <string>"FieldsV1", }, ], }, "spec": <map[string]interface {} | len:1>{ "gatewayRef": <map[string]interface {} | len:2>{ "name": <string>"maas-default-gateway", "namespace": <string>"openshift-ingress", }, }, "status": <map[string]interface {} | len:2>{ "conditions": <[]interface {} | len:5, cap:8>[ <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-07-15T12:33:55Z", "message": <string>"AuthConfig CRD (Kuadrant) is available on the cluster", "observedGeneration": <int64>1, "reason": <string>"DependenciesMet", "status": <string>"True", "type": <string>"DependenciesAvailable", }, <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-07-15T12:33:55Z", "message": <string>"All prerequisites are satisfied", "observedGeneration": <int64>1, "reason": <string>"PrerequisitesMet", "status": <string>"True", "type": <string>"MaaSPrerequisitesAvailable", }, <map[string]interface {} | len:6>{ "message": <string>"All prerequisites are satisfied", "observedGeneration": <int64>1, "reason": <string>"PrerequisitesMet", "status": <string>"False", "type": <string>"Degraded", "lastTransitionTime": <string>"2026-07-15T12:33:55Z", }, <map[string]interface {} | len:6>{ "reason": <string>"DeploymentsReady", "status": <string>"True", "type": <string>"DeploymentsAvailable", "lastTransitionTime": <string>"2026-07-15T12:34:30Z", "message": <string>"maas-api deployment is available", "observedGeneration": <int64>1, }, <map[string]interface {} | len:6>{ "message": <string>"MaaS platform manifests applied and maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"Reconciled", "status": <string>"True", "type": <string>"Ready", "lastTransitionTime": <string>"2026-07-15T12:34:30Z", }, ], "phase": <string>"Active", }, }, }, ] to be empty --- FAIL: TestOdhOperator/components/group_4/modelsasservice (527.25s) === RUN TestOdhOperator/components/group_4 --- FAIL: TestOdhOperator/components/group_4 (0.00s) === RUN TestOdhOperator/components 2026/07/15 12:41:49 TEST FAILURE DETECTED: group 4 2026/07/15 12:41:49 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 4) === 2026/07/15 12:41:51 === CLUSTER STATE === 2026/07/15 12:41:51 Node: ods-qe-psi-23-v5877-master-0 2026/07/15 12:41:51 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 12:41:51 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 12:41:51 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 12:41:51 Condition Ready: True - kubelet is posting ready status 2026/07/15 12:41:51 Resources: 11500m CPU, 23450692Ki memory allocatable (of 12000m CPU, 24601668Ki memory capacity) 2026/07/15 12:41:51 Node: ods-qe-psi-23-v5877-master-1 2026/07/15 12:41:51 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 12:41:51 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 12:41:51 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 12:41:51 Condition Ready: True - kubelet is posting ready status 2026/07/15 12:41:51 Resources: 11500m CPU, 23450696Ki memory allocatable (of 12000m CPU, 24601672Ki memory capacity) 2026/07/15 12:41:51 Node: ods-qe-psi-23-v5877-master-2 2026/07/15 12:41:51 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 12:41:51 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 12:41:51 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 12:41:51 Condition Ready: True - kubelet is posting ready status 2026/07/15 12:41:51 Resources: 11500m CPU, 23450696Ki memory allocatable (of 12000m CPU, 24601672Ki memory capacity) 2026/07/15 12:41:51 Node: ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/15 12:41:51 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 12:41:51 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 12:41:51 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 12:41:51 Condition Ready: True - kubelet is posting ready status 2026/07/15 12:41:51 Resources: 11500m CPU, 23450688Ki memory allocatable (of 12000m CPU, 24601664Ki memory capacity) 2026/07/15 12:41:51 Node: ods-qe-psi-23-v5877-worker-0-gdv9r 2026/07/15 12:41:51 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 12:41:51 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 12:41:51 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 12:41:51 Condition Ready: True - kubelet is posting ready status 2026/07/15 12:41:51 Resources: 11500m CPU, 23450696Ki memory allocatable (of 12000m CPU, 24601672Ki memory capacity) 2026/07/15 12:41:51 === DEPLOYMENTS === 2026/07/15 12:41:51 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/15 12:41:51 === PODS === 2026/07/15 12:41:51 Failed to collect pod data: redhat-ods-applications/maas-controller-85bdcf9988-5m2nb: container manager not ready 2026/07/15 12:41:51 === OPERATOR STATUS === 2026/07/15 12:41:51 Operator deployment: 3/3 ready 2026/07/15 12:41:51 === DSCI STATUS === 2026/07/15 12:41:51 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/15 12:41:51 === DSC STATUS === 2026/07/15 12:41:51 DSC default-dsc: 2026/07/15 12:41:51 Ready: True - 2026/07/15 12:41:51 ProvisioningSucceeded: True - 2026/07/15 12:41:51 ComponentsReady: True - 2026/07/15 12:41:51 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/15 12:41:51 DashboardReady: False - Component ManagementState is set to Removed 2026/07/15 12:41:51 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 12:41:51 KserveLLMInferenceServiceDependencies: True - 2026/07/15 12:41:51 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/15 12:41:51 KserveReady: True - 2026/07/15 12:41:51 KueueReady: False - Component ManagementState is set to Removed 2026/07/15 12:41:51 LLM-D-WVADependencies: True - 2026/07/15 12:41:51 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 12:41:51 ModelControllerReady: True - 2026/07/15 12:41:51 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/15 12:41:51 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/15 12:41:51 OGXReady: False - Component ManagementState is set to Removed 2026/07/15 12:41:51 RayReady: False - Component ManagementState is set to Removed 2026/07/15 12:41:51 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 12:41:51 TrainerReady: False - Component ManagementState is set to Removed 2026/07/15 12:41:51 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 12:41:51 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/15 12:41:51 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/15 12:41:51 === RECENT EVENTS === 2026/07/15 12:41:51 12:40:33 INFO Pod/maas-controller-85bdcf9988-5m2nb: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/15 12:41:51 12:40:33 INFO Pod/maas-controller-85bdcf9988-5m2nb: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 566ms (566ms including waiting). Image size: 212866077 bytes. 2026/07/15 12:41:51 12:38:30 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 12:41:51 12:38:05 WARN Pod/maas-controller-85bdcf9988-5m2nb: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-qhkdx); attempting to pull the image may not succeed. 2026/07/15 12:41:51 12:37:50 INFO Pod/maas-controller-85bdcf9988-5m2nb: Created - Created container: manager 2026/07/15 12:41:51 12:37:50 INFO Pod/maas-controller-85bdcf9988-5m2nb: Started - Started container manager 2026/07/15 12:41:51 12:37:50 INFO Pod/maas-controller-85bdcf9988-5m2nb: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 703ms (703ms including waiting). Image size: 212866077 bytes. 2026/07/15 12:41:51 === RESOURCE QUOTAS === 2026/07/15 12:41:51 No resource quotas found in monitored namespaces 2026/07/15 12:41:51 Classification for group 4: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/15 12:41:51 Evidence: container maas-controller-85bdcf9988-5m2nb/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-5m2nb_redhat-ods-applications(dcfe3957-cecb-4f62-84a7-ed4ded95adb7) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-85bdcf9988-5m2nb/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-5m2nb_redhat-ods-applications(dcfe3957-cecb-4f62-84a7-ed4ded95adb7)"],"confidence":"medium"} 2026/07/15 12:41:51 Diagnostics complete for failed test: group 4 controller_test.go:753: Stopping: group 4 test failed. 2026/07/15 12:41:51 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/15 12:41:51 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/15 12:43:03 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (1640.82s) === RUN TestOdhOperator 2026/07/15 12:16:43 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/15 12:44:16 TEST FAILURE DETECTED: components 2026/07/15 12:44:16 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/15 12:44:17 === CLUSTER STATE === 2026/07/15 12:44:17 Node: ods-qe-psi-23-v5877-master-0 2026/07/15 12:44:17 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 12:44:17 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 12:44:17 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 12:44:17 Condition Ready: True - kubelet is posting ready status 2026/07/15 12:44:17 Resources: 11500m CPU, 23450692Ki memory allocatable (of 12000m CPU, 24601668Ki memory capacity) 2026/07/15 12:44:17 Node: ods-qe-psi-23-v5877-master-1 2026/07/15 12:44:17 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 12:44:17 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 12:44:17 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 12:44:17 Condition Ready: True - kubelet is posting ready status 2026/07/15 12:44:17 Resources: 11500m CPU, 23450696Ki memory allocatable (of 12000m CPU, 24601672Ki memory capacity) 2026/07/15 12:44:17 Node: ods-qe-psi-23-v5877-master-2 2026/07/15 12:44:17 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 12:44:17 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 12:44:17 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 12:44:17 Condition Ready: True - kubelet is posting ready status 2026/07/15 12:44:17 Resources: 11500m CPU, 23450696Ki memory allocatable (of 12000m CPU, 24601672Ki memory capacity) 2026/07/15 12:44:17 Node: ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/15 12:44:17 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 12:44:17 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 12:44:17 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 12:44:17 Condition Ready: True - kubelet is posting ready status 2026/07/15 12:44:17 Resources: 11500m CPU, 23450688Ki memory allocatable (of 12000m CPU, 24601664Ki memory capacity) 2026/07/15 12:44:17 Node: ods-qe-psi-23-v5877-worker-0-gdv9r 2026/07/15 12:44:17 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 12:44:17 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 12:44:17 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 12:44:17 Condition Ready: True - kubelet is posting ready status 2026/07/15 12:44:17 Resources: 11500m CPU, 23450696Ki memory allocatable (of 12000m CPU, 24601672Ki memory capacity) 2026/07/15 12:44:17 === DEPLOYMENTS === 2026/07/15 12:44:17 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/15 12:44:17 === PODS === 2026/07/15 12:44:17 Failed to collect pod data: redhat-ods-applications/maas-controller-85bdcf9988-5m2nb: container manager not ready 2026/07/15 12:44:17 === OPERATOR STATUS === 2026/07/15 12:44:17 Operator deployment: 3/3 ready 2026/07/15 12:44:17 === DSCI STATUS === 2026/07/15 12:44:17 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/15 12:44:17 === DSC STATUS === 2026/07/15 12:44:17 DSC default-dsc: 2026/07/15 12:44:17 Ready: True - 2026/07/15 12:44:17 ProvisioningSucceeded: True - 2026/07/15 12:44:17 ComponentsReady: True - 2026/07/15 12:44:17 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/15 12:44:17 DashboardReady: False - Component ManagementState is set to Removed 2026/07/15 12:44:17 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 12:44:17 KserveLLMInferenceServiceDependencies: True - 2026/07/15 12:44:17 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/15 12:44:17 KserveReady: True - 2026/07/15 12:44:17 KueueReady: False - Component ManagementState is set to Removed 2026/07/15 12:44:17 LLM-D-WVADependencies: True - 2026/07/15 12:44:17 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 12:44:17 ModelControllerReady: True - 2026/07/15 12:44:17 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/15 12:44:17 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/15 12:44:17 OGXReady: False - Component ManagementState is set to Removed 2026/07/15 12:44:17 RayReady: False - Component ManagementState is set to Removed 2026/07/15 12:44:17 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 12:44:17 TrainerReady: True - 2026/07/15 12:44:17 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 12:44:17 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/15 12:44:17 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/15 12:44:17 === RECENT EVENTS === 2026/07/15 12:44:17 12:43:50 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-f2gvs_ede1913e-a14e-4d3e-9f26-1167fc55f95e became leader 2026/07/15 12:44:17 12:43:49 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-nlz7q_787961c9-bc00-4307-b5d1-ce4bc7476fb8 became leader 2026/07/15 12:44:17 12:43:32 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-r7t6b_1a66d663-70f1-4362-9a1d-46caa5931116 became leader 2026/07/15 12:44:17 12:43:31 INFO Pod/kserve-controller-manager-6c67dcb69c-nlz7q: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 641ms (641ms including waiting). Image size: 238435613 bytes. 2026/07/15 12:44:17 12:43:31 INFO Pod/llmisvc-controller-manager-5d65df97bb-f2gvs: Started - Started container manager 2026/07/15 12:44:17 12:43:31 INFO Pod/llmisvc-controller-manager-5d65df97bb-f2gvs: Created - Created container: manager 2026/07/15 12:44:17 12:43:31 INFO Pod/llmisvc-controller-manager-5d65df97bb-f2gvs: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 414ms (414ms including waiting). Image size: 236366599 bytes. 2026/07/15 12:44:17 12:43:31 INFO Pod/kserve-controller-manager-6c67dcb69c-nlz7q: Started - Started container manager 2026/07/15 12:44:17 12:43:31 INFO Pod/kserve-controller-manager-6c67dcb69c-nlz7q: Created - Created container: manager 2026/07/15 12:44:17 12:43:30 INFO Pod/kserve-controller-manager-6c67dcb69c-nlz7q: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/15 12:44:17 12:43:30 INFO Pod/kserve-controller-manager-6c67dcb69c-nlz7q: AddedInterface - Add eth0 [10.128.2.124/23] from ovn-kubernetes 2026/07/15 12:44:17 12:43:30 INFO Pod/llmisvc-controller-manager-5d65df97bb-f2gvs: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/15 12:44:17 12:43:30 INFO Pod/llmisvc-controller-manager-5d65df97bb-f2gvs: AddedInterface - Add eth0 [10.128.2.125/23] from ovn-kubernetes 2026/07/15 12:44:17 12:43:29 WARN Service/llmisvc-controller-manager-service: ClusterIPNotAllocated - Cluster IP [IPv4]: 172.30.57.117 is not allocated; repairing 2026/07/15 12:44:17 12:43:29 WARN Service/kserve-controller-manager-service: ClusterIPNotAllocated - Cluster IP [IPv4]: 172.30.81.0 is not allocated; repairing 2026/07/15 12:44:17 12:43:29 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/15 12:44:17 12:43:29 INFO Pod/kserve-controller-manager-6c67dcb69c-nlz7q: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-nlz7q to ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/15 12:44:17 12:43:29 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-nlz7q 2026/07/15 12:44:17 12:43:29 INFO Pod/llmisvc-controller-manager-5d65df97bb-f2gvs: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-f2gvs to ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/15 12:44:17 12:43:29 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-f2gvs 2026/07/15 12:44:17 12:43:29 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/15 12:44:17 12:43:16 INFO Pod/model-serving-api-6688cd56bd-gdhst: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/15 12:44:17 12:43:16 INFO Pod/odh-model-controller-6849fc49d8-r7t6b: Started - Started container manager 2026/07/15 12:44:17 12:43:16 INFO Pod/model-serving-api-6688cd56bd-gdhst: Started - Started container server 2026/07/15 12:44:17 12:43:16 INFO Pod/model-serving-api-6688cd56bd-gdhst: Created - Created container: server 2026/07/15 12:44:17 12:43:16 INFO Pod/odh-model-controller-6849fc49d8-r7t6b: Created - Created container: manager 2026/07/15 12:44:17 12:43:16 INFO Pod/odh-model-controller-6849fc49d8-r7t6b: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 511ms (511ms including waiting). Image size: 238959091 bytes. 2026/07/15 12:44:17 12:43:16 INFO Pod/model-serving-api-6688cd56bd-gdhst: AddedInterface - Add eth0 [10.128.2.123/23] from ovn-kubernetes 2026/07/15 12:44:17 12:43:15 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/15 12:44:17 12:43:15 INFO Pod/model-serving-api-6688cd56bd-gdhst: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-gdhst to ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/15 12:44:17 12:43:15 INFO Pod/odh-model-controller-6849fc49d8-r7t6b: AddedInterface - Add eth0 [10.128.2.122/23] from ovn-kubernetes 2026/07/15 12:44:17 12:43:15 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-gdhst 2026/07/15 12:44:17 12:43:15 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/15 12:44:17 12:43:15 INFO Pod/odh-model-controller-6849fc49d8-r7t6b: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/15 12:44:17 12:43:15 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 12:44:17 12:43:14 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-r7t6b 2026/07/15 12:44:17 12:43:14 WARN Pod/odh-model-controller-6849fc49d8-r7t6b: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/15 12:44:17 12:43:14 INFO Pod/odh-model-controller-6849fc49d8-r7t6b: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-r7t6b to ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/15 12:44:17 12:43:14 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/15 12:44:17 12:43:07 WARN Pod/maas-controller-85bdcf9988-5m2nb: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-qhkdx); attempting to pull the image may not succeed. 2026/07/15 12:44:17 12:43:04 INFO Pod/kserve-controller-manager-6c67dcb69c-9jlfr: Killing - Stopping container manager 2026/07/15 12:44:17 12:43:04 INFO Pod/model-serving-api-6688cd56bd-fp9vg: Killing - Stopping container server 2026/07/15 12:44:17 12:43:04 INFO Pod/odh-model-controller-6849fc49d8-cmh8s: Killing - Stopping container manager 2026/07/15 12:44:17 12:43:04 INFO Pod/llmisvc-controller-manager-5d65df97bb-xdjdf: Killing - Stopping container manager 2026/07/15 12:44:17 12:43:04 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 12:44:17 12:42:43 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-xdjdf_6ddfbbcf-d38f-4ff7-8a8d-09e80ef2c5ac became leader 2026/07/15 12:44:17 12:42:39 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-9jlfr_44e9e461-31f0-46f1-84f5-67cd48289817 became leader 2026/07/15 12:44:17 12:42:26 INFO Pod/llmisvc-controller-manager-5d65df97bb-xdjdf: Started - Started container manager 2026/07/15 12:44:17 12:42:26 INFO Pod/llmisvc-controller-manager-5d65df97bb-xdjdf: Created - Created container: manager 2026/07/15 12:44:17 12:42:26 INFO Pod/llmisvc-controller-manager-5d65df97bb-xdjdf: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 2.888s (2.888s including waiting). Image size: 236366599 bytes. 2026/07/15 12:44:17 12:42:24 INFO Pod/kserve-controller-manager-6c67dcb69c-9jlfr: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 529ms (529ms including waiting). Image size: 238435613 bytes. 2026/07/15 12:44:17 12:42:24 INFO Pod/kserve-controller-manager-6c67dcb69c-9jlfr: Created - Created container: manager 2026/07/15 12:44:17 12:42:24 INFO Pod/kserve-controller-manager-6c67dcb69c-9jlfr: Started - Started container manager 2026/07/15 12:44:17 12:42:23 INFO Pod/llmisvc-controller-manager-5d65df97bb-xdjdf: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-xdjdf to ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/15 12:44:17 12:42:23 INFO Pod/kserve-controller-manager-6c67dcb69c-9jlfr: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/15 12:44:17 12:42:23 INFO Pod/kserve-controller-manager-6c67dcb69c-9jlfr: AddedInterface - Add eth0 [10.128.2.121/23] from ovn-kubernetes 2026/07/15 12:44:17 12:42:23 INFO Pod/kserve-controller-manager-6c67dcb69c-9jlfr: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-9jlfr to ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/15 12:44:17 12:42:23 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-9jlfr 2026/07/15 12:44:17 12:42:23 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/15 12:44:17 12:42:23 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/15 12:44:17 12:42:23 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-xdjdf 2026/07/15 12:44:17 12:42:23 INFO Pod/llmisvc-controller-manager-5d65df97bb-xdjdf: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/15 12:44:17 12:42:23 INFO Pod/llmisvc-controller-manager-5d65df97bb-xdjdf: AddedInterface - Add eth0 [10.128.2.120/23] from ovn-kubernetes 2026/07/15 12:44:17 12:42:22 WARN Service/kserve-controller-manager-service: ClusterIPNotAllocated - Cluster IP [IPv4]: 172.30.17.218 is not allocated; repairing 2026/07/15 12:44:17 12:42:20 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-cmh8s_239abd4a-3ff9-4a73-96e8-48187ec03330 became leader 2026/07/15 12:44:17 12:42:14 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-7ccfdd66-nnvq7_d097566c-6c9f-474b-82e7-6c1651d86d2e became leader 2026/07/15 12:44:17 12:42:05 INFO Pod/model-serving-api-6688cd56bd-fp9vg: Created - Created container: server 2026/07/15 12:44:17 12:42:05 INFO Pod/model-serving-api-6688cd56bd-fp9vg: Started - Started container server 2026/07/15 12:44:17 12:42:05 INFO Pod/model-serving-api-6688cd56bd-fp9vg: AddedInterface - Add eth0 [10.128.2.118/23] from ovn-kubernetes 2026/07/15 12:44:17 12:42:05 INFO Pod/odh-model-controller-6849fc49d8-cmh8s: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 508ms (508ms including waiting). Image size: 238959091 bytes. 2026/07/15 12:44:17 12:42:05 INFO Pod/odh-model-controller-6849fc49d8-cmh8s: Created - Created container: manager 2026/07/15 12:44:17 12:42:05 INFO Pod/odh-model-controller-6849fc49d8-cmh8s: Started - Started container manager 2026/07/15 12:44:17 12:42:05 INFO Pod/model-serving-api-6688cd56bd-fp9vg: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/15 12:44:17 12:42:04 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-fp9vg 2026/07/15 12:44:17 12:42:04 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/15 12:44:17 12:42:04 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/15 12:44:17 12:42:04 INFO Pod/model-serving-api-6688cd56bd-fp9vg: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-fp9vg to ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/15 12:44:17 12:42:04 INFO Pod/odh-model-controller-6849fc49d8-cmh8s: AddedInterface - Add eth0 [10.128.2.117/23] from ovn-kubernetes 2026/07/15 12:44:17 12:42:04 INFO Pod/odh-model-controller-6849fc49d8-cmh8s: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/15 12:44:17 12:42:03 INFO Pod/odh-model-controller-6849fc49d8-cmh8s: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-cmh8s to ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/15 12:44:17 12:42:03 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-cmh8s 2026/07/15 12:44:17 12:42:03 WARN Pod/odh-model-controller-6849fc49d8-cmh8s: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/15 12:44:17 12:42:03 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/15 12:44:17 12:41:56 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-nnvq7: AddedInterface - Add eth0 [10.131.1.45/23] from ovn-kubernetes 2026/07/15 12:44:17 12:41:56 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-nnvq7: Pulled - Container image "registry.redhat.io/rhoai/odh-trainer-rhel9@sha256:59ea523e2df9c1e705c755308ac11420682081cc63d54890e2b15fbd7d8ba570" already present on machine 2026/07/15 12:44:17 12:41:56 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-nnvq7: Created - Created container: manager 2026/07/15 12:44:17 12:41:56 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-nnvq7: Started - Started container manager 2026/07/15 12:44:17 12:41:55 INFO ReplicaSet/kubeflow-trainer-controller-manager-7ccfdd66: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-7ccfdd66-nnvq7 2026/07/15 12:44:17 12:41:55 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-7ccfdd66 from 0 to 1 2026/07/15 12:44:17 12:41:55 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-nnvq7: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-7ccfdd66-nnvq7 to ods-qe-psi-23-v5877-worker-0-gdv9r 2026/07/15 12:44:17 12:41:52 INFO Pod/odh-model-controller-6849fc49d8-qgdx5: Killing - Stopping container manager 2026/07/15 12:44:17 12:41:52 INFO Pod/llmisvc-controller-manager-5d65df97bb-v2bpc: Killing - Stopping container manager 2026/07/15 12:44:17 12:41:52 INFO Pod/model-serving-api-6688cd56bd-m8jqb: Killing - Stopping container server 2026/07/15 12:44:17 12:41:52 INFO Pod/kserve-controller-manager-6c67dcb69c-m845d: Killing - Stopping container manager 2026/07/15 12:44:17 12:40:33 INFO Pod/maas-controller-85bdcf9988-5m2nb: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/15 12:44:17 12:40:33 INFO Pod/maas-controller-85bdcf9988-5m2nb: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 566ms (566ms including waiting). Image size: 212866077 bytes. 2026/07/15 12:44:17 === RESOURCE QUOTAS === 2026/07/15 12:44:17 No resource quotas found in monitored namespaces 2026/07/15 12:44:17 Classification for components: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/15 12:44:17 Evidence: container maas-controller-85bdcf9988-5m2nb/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-5m2nb_redhat-ods-applications(dcfe3957-cecb-4f62-84a7-ed4ded95adb7) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-85bdcf9988-5m2nb/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-5m2nb_redhat-ods-applications(dcfe3957-cecb-4f62-84a7-ed4ded95adb7)"],"confidence":"medium"} 2026/07/15 12:44:17 Diagnostics complete for failed test: components controller_test.go:484: Stopping: components test failed. 2026/07/15 12:44:17 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/15 12:44:17 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/15 12:44:21 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (1657.98s)