=== RUN TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable helper_test.go:150: Verifying Tenant models-as-a-service/default-tenant is present before disable helper_test.go:150: Disabling MaaS subcomponent (setting to Removed) helper_test.go:150: Waiting for Tenant models-as-a-service/default-tenant to be deleted testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable (430.16s) === RUN TestOdhOperator/components/group_4/modelsasservice === PAUSE TestOdhOperator/components/group_4/modelsasservice === CONT TestOdhOperator/components/group_4/modelsasservice 2026-07-14T16:17:32Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.20.14"}} controller_test.go:753: Creating MaaS PostgreSQL instance in namespace redhat-ods-applications controller_test.go:753: MaaS PostgreSQL instance and maas-db-config secret created in namespace redhat-ods-applications controller_test.go:753: Creating MaaS GatewayClass and Gateway: openshift-ingress/maas-default-gateway controller_test.go:753: Using hostname=maas.apps.ods-qe-psi-23.osp.rh-ods.com, certSecret=router-certs-default controller_test.go:753: MaaS Gateway openshift-ingress/maas-default-gateway created with HTTP+HTTPS listeners test_context_test.go:615: Timed out after 420.008s. The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1824 with: Expected resource list 'models-as-a-service/default-tenant' of kind 'Tenant' to be empty, but it contains resources. Expected <[]unstructured.Unstructured | len:1, cap:1>: [ { Object: { "kind": <string>"Tenant", "metadata": <map[string]interface {} | len:8>{ "generation": <int64>1, "managedFields": <[]interface {} | len:2, cap:2>[ <map[string]interface {} | len:6>{ "time": <string>"2026-07-14T16:18:23Z", "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:2>{ "f:metadata": <map[string]interface {} | len:1>{ "f:ownerReferences": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "k:{\"uid\":\"a4d71f2d-76f2-4622-b8f8-d96493e74a30\"}": <map[string]interface {} | len:0>{}, }, }, "f:spec": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "f:gatewayRef": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:name": <map[string]interface {} | len:0>{}, "f:namespace": <map[string]interface {} | len:0>{}, }, }, }, "manager": <string>"manager", "operation": <string>"Update", }, <map[string]interface {} | len:7>{ "time": <string>"2026-07-14T16:18:48Z", "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:1>{ "f:status": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:conditions": <map[string]interface {} | len:0>{}, "f:phase": <map[string]interface {} | len:0>{}, }, }, "manager": <string>"manager", "operation": <string>"Update", "subresource": <string>"status", }, ], "name": <string>"default-tenant", "namespace": <string>"models-as-a-service", "ownerReferences": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:4>{ "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "kind": <string>"Config", "name": <string>"default", "uid": <string>"a4d71f2d-76f2-4622-b8f8-d96493e74a30", }, ], "resourceVersion": <string>"267180016", "uid": <string>"1bf145ce-cf8f-4765-97f8-ae8cd4dbbc97", "creationTimestamp": <string>"2026-07-14T16:18:23Z", }, "spec": <map[string]interface {} | len:1>{ "gatewayRef": <map[string]interface {} | len:2>{ "name": <string>"maas-default-gateway", "namespace": <string>"openshift-ingress", }, }, "status": <map[string]interface {} | len:2>{ "conditions": <[]interface {} | len:5, cap:8>[ <map[string]interface {} | len:6>{ "status": <string>"True", "type": <string>"DependenciesAvailable", "lastTransitionTime": <string>"2026-07-14T16:18:27Z", "message": <string>"AuthConfig CRD (Kuadrant) is available on the cluster", "observedGeneration": <int64>1, "reason": <string>"DependenciesMet", }, <map[string]interface {} | len:6>{ "status": <string>"True", "type": <string>"MaaSPrerequisitesAvailable", "lastTransitionTime": <string>"2026-07-14T16:18:27Z", "message": <string>"All prerequisites are satisfied", "observedGeneration": <int64>1, "reason": <string>"PrerequisitesMet", }, <map[string]interface {} | len:6>{ "status": <string>"False", "type": <string>"Degraded", "lastTransitionTime": <string>"2026-07-14T16:18:27Z", "message": <string>"All prerequisites are satisfied", "observedGeneration": <int64>1, "reason": <string>"PrerequisitesMet", }, <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-07-14T16:18:48Z", "message": <string>"maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"DeploymentsReady", "status": <string>"True", "type": <string>"DeploymentsAvailable", }, <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-07-14T16:18:48Z", "message": <string>"MaaS platform manifests applied and maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"Reconciled", "status": <string>"True", "type": <string>"Ready", }, ], "phase": <string>"Active", }, "apiVersion": <string>"maas.opendatahub.io/v1alpha1", }, }, ] to be empty --- FAIL: TestOdhOperator/components/group_4/modelsasservice (512.54s) === RUN TestOdhOperator/components/group_4 --- FAIL: TestOdhOperator/components/group_4 (0.00s) === RUN TestOdhOperator/components 2026/07/14 16:26:04 TEST FAILURE DETECTED: group 4 2026/07/14 16:26:04 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 4) === 2026/07/14 16:26:06 === CLUSTER STATE === 2026/07/14 16:26:06 Node: ods-qe-psi-23-v5877-master-0 2026/07/14 16:26:06 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/14 16:26:06 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/14 16:26:06 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/14 16:26:06 Condition Ready: True - kubelet is posting ready status 2026/07/14 16:26:06 Resources: 11500m CPU, 23450692Ki memory allocatable (of 12000m CPU, 24601668Ki memory capacity) 2026/07/14 16:26:06 Node: ods-qe-psi-23-v5877-master-1 2026/07/14 16:26:06 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/14 16:26:06 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/14 16:26:06 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/14 16:26:06 Condition Ready: True - kubelet is posting ready status 2026/07/14 16:26:06 Resources: 11500m CPU, 23450696Ki memory allocatable (of 12000m CPU, 24601672Ki memory capacity) 2026/07/14 16:26:06 Node: ods-qe-psi-23-v5877-master-2 2026/07/14 16:26:06 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/14 16:26:06 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/14 16:26:06 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/14 16:26:06 Condition Ready: True - kubelet is posting ready status 2026/07/14 16:26:06 Resources: 11500m CPU, 23450696Ki memory allocatable (of 12000m CPU, 24601672Ki memory capacity) 2026/07/14 16:26:06 Node: ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/14 16:26:06 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/14 16:26:06 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/14 16:26:06 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/14 16:26:06 Condition Ready: True - kubelet is posting ready status 2026/07/14 16:26:06 Resources: 11500m CPU, 23450688Ki memory allocatable (of 12000m CPU, 24601664Ki memory capacity) 2026/07/14 16:26:06 Node: ods-qe-psi-23-v5877-worker-0-gdv9r 2026/07/14 16:26:06 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/14 16:26:06 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/14 16:26:06 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/14 16:26:06 Condition Ready: True - kubelet is posting ready status 2026/07/14 16:26:06 Resources: 11500m CPU, 23450696Ki memory allocatable (of 12000m CPU, 24601672Ki memory capacity) 2026/07/14 16:26:06 === DEPLOYMENTS === 2026/07/14 16:26:06 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/14 16:26:06 === PODS === 2026/07/14 16:26:06 Failed to collect pod data: redhat-ods-applications/maas-controller-85bdcf9988-hmmld: container manager not ready 2026/07/14 16:26:06 === OPERATOR STATUS === 2026/07/14 16:26:06 Operator deployment: 3/3 ready 2026/07/14 16:26:06 === DSCI STATUS === 2026/07/14 16:26:06 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/14 16:26:06 === DSC STATUS === 2026/07/14 16:26:06 DSC default-dsc: 2026/07/14 16:26:06 Ready: True - 2026/07/14 16:26:06 ProvisioningSucceeded: True - 2026/07/14 16:26:06 ComponentsReady: True - 2026/07/14 16:26:06 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/14 16:26:06 DashboardReady: False - Component ManagementState is set to Removed 2026/07/14 16:26:06 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/14 16:26:06 KserveLLMInferenceServiceDependencies: True - 2026/07/14 16:26:06 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/14 16:26:06 KserveReady: True - 2026/07/14 16:26:06 KueueReady: False - Component ManagementState is set to Removed 2026/07/14 16:26:06 LLM-D-WVADependencies: True - 2026/07/14 16:26:06 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/14 16:26:06 ModelControllerReady: True - 2026/07/14 16:26:06 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/14 16:26:06 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/14 16:26:06 OGXReady: False - Component ManagementState is set to Removed 2026/07/14 16:26:06 RayReady: False - Component ManagementState is set to Removed 2026/07/14 16:26:06 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/14 16:26:06 TrainerReady: False - Component ManagementState is set to Removed 2026/07/14 16:26:06 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/14 16:26:06 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/14 16:26:06 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/14 16:26:06 === RECENT EVENTS === 2026/07/14 16:26:06 16:24:59 INFO Pod/maas-controller-85bdcf9988-hmmld: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 1.269s (1.269s including waiting). Image size: 212866077 bytes. 2026/07/14 16:26:06 16:24:58 INFO Pod/maas-controller-85bdcf9988-hmmld: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/14 16:26:06 16:24:06 WARN Pod/maas-controller-85bdcf9988-hmmld: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-njx76); attempting to pull the image may not succeed. 2026/07/14 16:26:06 16:24:03 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/14 16:26:06 16:22:17 INFO Pod/maas-controller-85bdcf9988-hmmld: Created - Created container: manager 2026/07/14 16:26:06 16:22:17 INFO Pod/maas-controller-85bdcf9988-hmmld: Started - Started container manager 2026/07/14 16:26:06 16:22:17 INFO Pod/maas-controller-85bdcf9988-hmmld: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 677ms (677ms including waiting). Image size: 212866077 bytes. 2026/07/14 16:26:06 === RESOURCE QUOTAS === 2026/07/14 16:26:06 No resource quotas found in monitored namespaces 2026/07/14 16:26:06 Classification for group 4: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/14 16:26:06 Evidence: container maas-controller-85bdcf9988-hmmld/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-hmmld_redhat-ods-applications(eb60e58f-4f60-40cf-9cbd-1c5958018d9b) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-85bdcf9988-hmmld/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-hmmld_redhat-ods-applications(eb60e58f-4f60-40cf-9cbd-1c5958018d9b)"],"confidence":"medium"} 2026/07/14 16:26:06 Diagnostics complete for failed test: group 4 controller_test.go:753: Stopping: group 4 test failed. 2026/07/14 16:26:06 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/14 16:26:06 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/14 16:27:17 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (1498.87s) === RUN TestOdhOperator 2026/07/14 16:03:19 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/14 16:28:29 TEST FAILURE DETECTED: components 2026/07/14 16:28:29 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/14 16:28:30 === CLUSTER STATE === 2026/07/14 16:28:30 Node: ods-qe-psi-23-v5877-master-0 2026/07/14 16:28:30 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/14 16:28:30 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/14 16:28:30 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/14 16:28:30 Condition Ready: True - kubelet is posting ready status 2026/07/14 16:28:30 Resources: 11500m CPU, 23450692Ki memory allocatable (of 12000m CPU, 24601668Ki memory capacity) 2026/07/14 16:28:30 Node: ods-qe-psi-23-v5877-master-1 2026/07/14 16:28:30 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/14 16:28:30 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/14 16:28:30 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/14 16:28:30 Condition Ready: True - kubelet is posting ready status 2026/07/14 16:28:30 Resources: 11500m CPU, 23450696Ki memory allocatable (of 12000m CPU, 24601672Ki memory capacity) 2026/07/14 16:28:30 Node: ods-qe-psi-23-v5877-master-2 2026/07/14 16:28:30 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/14 16:28:30 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/14 16:28:30 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/14 16:28:30 Condition Ready: True - kubelet is posting ready status 2026/07/14 16:28:30 Resources: 11500m CPU, 23450696Ki memory allocatable (of 12000m CPU, 24601672Ki memory capacity) 2026/07/14 16:28:30 Node: ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/14 16:28:30 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/14 16:28:30 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/14 16:28:30 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/14 16:28:30 Condition Ready: True - kubelet is posting ready status 2026/07/14 16:28:30 Resources: 11500m CPU, 23450688Ki memory allocatable (of 12000m CPU, 24601664Ki memory capacity) 2026/07/14 16:28:30 Node: ods-qe-psi-23-v5877-worker-0-gdv9r 2026/07/14 16:28:30 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/14 16:28:30 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/14 16:28:30 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/14 16:28:30 Condition Ready: True - kubelet is posting ready status 2026/07/14 16:28:30 Resources: 11500m CPU, 23450696Ki memory allocatable (of 12000m CPU, 24601672Ki memory capacity) 2026/07/14 16:28:30 === DEPLOYMENTS === 2026/07/14 16:28:30 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/14 16:28:30 === PODS === 2026/07/14 16:28:30 Failed to collect pod data: redhat-ods-applications/maas-controller-85bdcf9988-hmmld: container manager not ready 2026/07/14 16:28:30 === OPERATOR STATUS === 2026/07/14 16:28:30 Operator deployment: 3/3 ready 2026/07/14 16:28:30 === DSCI STATUS === 2026/07/14 16:28:30 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/14 16:28:30 === DSC STATUS === 2026/07/14 16:28:30 DSC default-dsc: 2026/07/14 16:28:30 Ready: True - 2026/07/14 16:28:30 ProvisioningSucceeded: True - 2026/07/14 16:28:30 ComponentsReady: True - 2026/07/14 16:28:30 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/14 16:28:30 DashboardReady: False - Component ManagementState is set to Removed 2026/07/14 16:28:30 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/14 16:28:30 KserveLLMInferenceServiceDependencies: True - 2026/07/14 16:28:30 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/14 16:28:30 KserveReady: True - 2026/07/14 16:28:30 KueueReady: False - Component ManagementState is set to Removed 2026/07/14 16:28:30 LLM-D-WVADependencies: True - 2026/07/14 16:28:30 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/14 16:28:30 ModelControllerReady: True - 2026/07/14 16:28:30 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/14 16:28:30 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/14 16:28:30 OGXReady: False - Component ManagementState is set to Removed 2026/07/14 16:28:30 RayReady: False - Component ManagementState is set to Removed 2026/07/14 16:28:30 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/14 16:28:30 TrainerReady: True - 2026/07/14 16:28:30 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/14 16:28:30 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/14 16:28:30 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/14 16:28:30 === RECENT EVENTS === 2026/07/14 16:28:30 16:28:07 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-4c7zf_e168b6a9-7131-4c48-a59d-d4f5b5021c23 became leader 2026/07/14 16:28:30 16:28:06 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-9rztc_a77c8d4a-455a-49fd-b9a1-ab90f38a9608 became leader 2026/07/14 16:28:30 16:27:50 INFO Pod/kserve-controller-manager-6c67dcb69c-9rztc: Started - Started container manager 2026/07/14 16:28:30 16:27:50 INFO Pod/llmisvc-controller-manager-5d65df97bb-4c7zf: Started - Started container manager 2026/07/14 16:28:30 16:27:50 INFO Pod/kserve-controller-manager-6c67dcb69c-9rztc: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 519ms (519ms including waiting). Image size: 238435613 bytes. 2026/07/14 16:28:30 16:27:50 INFO Pod/kserve-controller-manager-6c67dcb69c-9rztc: Created - Created container: manager 2026/07/14 16:28:30 16:27:50 INFO Pod/kserve-controller-manager-6c67dcb69c-9rztc: AddedInterface - Add eth0 [10.128.3.59/23] from ovn-kubernetes 2026/07/14 16:28:30 16:27:50 INFO Pod/llmisvc-controller-manager-5d65df97bb-4c7zf: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 526ms (526ms including waiting). Image size: 236366599 bytes. 2026/07/14 16:28:30 16:27:50 INFO Pod/llmisvc-controller-manager-5d65df97bb-4c7zf: Created - Created container: manager 2026/07/14 16:28:30 16:27:50 INFO Pod/kserve-controller-manager-6c67dcb69c-9rztc: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/14 16:28:30 16:27:49 WARN Pod/kserve-controller-manager-6c67dcb69c-9rztc: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 2026/07/14 16:28:30 16:27:49 INFO Pod/llmisvc-controller-manager-5d65df97bb-4c7zf: AddedInterface - Add eth0 [10.128.3.60/23] from ovn-kubernetes 2026/07/14 16:28:30 16:27:49 INFO Pod/llmisvc-controller-manager-5d65df97bb-4c7zf: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/14 16:28:30 16:27:48 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-4l8f4_51da874e-a409-4134-a36d-9e5d74a1190e became leader 2026/07/14 16:28:30 16:27:48 INFO Pod/llmisvc-controller-manager-5d65df97bb-4c7zf: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-4c7zf to ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/14 16:28:30 16:27:48 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-9rztc 2026/07/14 16:28:30 16:27:48 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/14 16:28:30 16:27:48 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-4c7zf 2026/07/14 16:28:30 16:27:48 INFO Pod/kserve-controller-manager-6c67dcb69c-9rztc: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-9rztc to ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/14 16:28:30 16:27:48 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/14 16:28:30 16:27:32 INFO Pod/odh-model-controller-6849fc49d8-4l8f4: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 838ms (838ms including waiting). Image size: 238959091 bytes. 2026/07/14 16:28:30 16:27:32 INFO Pod/odh-model-controller-6849fc49d8-4l8f4: Started - Started container manager 2026/07/14 16:28:30 16:27:32 INFO Pod/odh-model-controller-6849fc49d8-4l8f4: Created - Created container: manager 2026/07/14 16:28:30 16:27:32 INFO Pod/model-serving-api-6688cd56bd-7dtzj: AddedInterface - Add eth0 [10.128.3.58/23] from ovn-kubernetes 2026/07/14 16:28:30 16:27:32 INFO Pod/model-serving-api-6688cd56bd-7dtzj: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/14 16:28:30 16:27:32 INFO Pod/model-serving-api-6688cd56bd-7dtzj: Created - Created container: server 2026/07/14 16:28:30 16:27:32 INFO Pod/model-serving-api-6688cd56bd-7dtzj: Started - Started container server 2026/07/14 16:28:30 16:27:31 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/14 16:28:30 16:27:31 INFO Pod/model-serving-api-6688cd56bd-7dtzj: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-7dtzj to ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/14 16:28:30 16:27:31 INFO Pod/odh-model-controller-6849fc49d8-4l8f4: AddedInterface - Add eth0 [10.128.3.57/23] from ovn-kubernetes 2026/07/14 16:28:30 16:27:31 INFO Pod/odh-model-controller-6849fc49d8-4l8f4: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/14 16:28:30 16:27:31 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/14 16:28:30 16:27:31 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/14 16:28:30 16:27:31 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-7dtzj 2026/07/14 16:28:30 16:27:31 WARN Pod/model-serving-api-6688cd56bd-7dtzj: FailedMount - MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 2026/07/14 16:28:30 16:27:30 WARN Pod/odh-model-controller-6849fc49d8-4l8f4: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/14 16:28:30 16:27:29 INFO Pod/odh-model-controller-6849fc49d8-4l8f4: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-4l8f4 to ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/14 16:28:30 16:27:29 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/14 16:28:30 16:27:29 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-4l8f4 2026/07/14 16:28:30 16:27:19 INFO Pod/odh-model-controller-6849fc49d8-ndthx: Killing - Stopping container manager 2026/07/14 16:28:30 16:27:19 INFO Pod/model-serving-api-6688cd56bd-2xz94: Killing - Stopping container server 2026/07/14 16:28:30 16:27:18 INFO Pod/llmisvc-controller-manager-5d65df97bb-fmpvj: Killing - Stopping container manager 2026/07/14 16:28:30 16:27:18 INFO Pod/kserve-controller-manager-6c67dcb69c-rr2nc: Killing - Stopping container manager 2026/07/14 16:28:30 16:26:54 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-fmpvj_100ceb9b-5bce-43e1-aa23-730b02eacfa1 became leader 2026/07/14 16:28:30 16:26:49 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-rr2nc_1d8deb64-3f82-437a-b6a2-206575c49950 became leader 2026/07/14 16:28:30 16:26:37 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-ndthx_973c599e-7851-4b3f-91a8-84995fc7af65 became leader 2026/07/14 16:28:30 16:26:34 INFO Pod/llmisvc-controller-manager-5d65df97bb-fmpvj: Created - Created container: manager 2026/07/14 16:28:30 16:26:34 INFO Pod/llmisvc-controller-manager-5d65df97bb-fmpvj: Started - Started container manager 2026/07/14 16:28:30 16:26:33 INFO Pod/kserve-controller-manager-6c67dcb69c-rr2nc: Created - Created container: manager 2026/07/14 16:28:30 16:26:33 INFO Pod/llmisvc-controller-manager-5d65df97bb-fmpvj: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 493ms (494ms including waiting). Image size: 236366599 bytes. 2026/07/14 16:28:30 16:26:33 INFO Pod/llmisvc-controller-manager-5d65df97bb-fmpvj: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/14 16:28:30 16:26:33 INFO Pod/llmisvc-controller-manager-5d65df97bb-fmpvj: AddedInterface - Add eth0 [10.128.3.56/23] from ovn-kubernetes 2026/07/14 16:28:30 16:26:33 INFO Pod/kserve-controller-manager-6c67dcb69c-rr2nc: Started - Started container manager 2026/07/14 16:28:30 16:26:33 INFO Pod/kserve-controller-manager-6c67dcb69c-rr2nc: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 802ms (802ms including waiting). Image size: 238435613 bytes. 2026/07/14 16:28:30 16:26:32 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/14 16:28:30 16:26:32 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-fmpvj 2026/07/14 16:28:30 16:26:32 INFO Pod/kserve-controller-manager-6c67dcb69c-rr2nc: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-rr2nc to ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/14 16:28:30 16:26:32 INFO Pod/kserve-controller-manager-6c67dcb69c-rr2nc: AddedInterface - Add eth0 [10.128.3.55/23] from ovn-kubernetes 2026/07/14 16:28:30 16:26:32 INFO Pod/kserve-controller-manager-6c67dcb69c-rr2nc: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/14 16:28:30 16:26:32 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-rr2nc 2026/07/14 16:28:30 16:26:32 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/14 16:28:30 16:26:32 INFO Pod/llmisvc-controller-manager-5d65df97bb-fmpvj: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-fmpvj to ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/14 16:28:30 16:26:32 WARN Pod/llmisvc-controller-manager-5d65df97bb-fmpvj: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 2026/07/14 16:28:30 16:26:28 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-7ccfdd66-hs4lj_d45aed41-0856-4afd-b586-8b3f3fcc8d5e became leader 2026/07/14 16:28:30 16:26:21 INFO Pod/odh-model-controller-6849fc49d8-ndthx: Created - Created container: manager 2026/07/14 16:28:30 16:26:21 INFO Pod/model-serving-api-6688cd56bd-2xz94: Started - Started container server 2026/07/14 16:28:30 16:26:21 INFO Pod/model-serving-api-6688cd56bd-2xz94: AddedInterface - Add eth0 [10.128.3.54/23] from ovn-kubernetes 2026/07/14 16:28:30 16:26:21 INFO Pod/odh-model-controller-6849fc49d8-ndthx: Started - Started container manager 2026/07/14 16:28:30 16:26:21 INFO Pod/model-serving-api-6688cd56bd-2xz94: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/14 16:28:30 16:26:21 INFO Pod/model-serving-api-6688cd56bd-2xz94: Created - Created container: server 2026/07/14 16:28:30 16:26:20 INFO Pod/odh-model-controller-6849fc49d8-ndthx: AddedInterface - Add eth0 [10.128.3.53/23] from ovn-kubernetes 2026/07/14 16:28:30 16:26:20 INFO Pod/odh-model-controller-6849fc49d8-ndthx: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 601ms (601ms including waiting). Image size: 238959091 bytes. 2026/07/14 16:28:30 16:26:20 INFO Pod/odh-model-controller-6849fc49d8-ndthx: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/14 16:28:30 16:26:19 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/14 16:28:30 16:26:19 WARN Pod/model-serving-api-6688cd56bd-2xz94: FailedMount - MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 2026/07/14 16:28:30 16:26:19 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/14 16:28:30 16:26:19 INFO Pod/model-serving-api-6688cd56bd-2xz94: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-2xz94 to ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/14 16:28:30 16:26:19 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-2xz94 2026/07/14 16:28:30 16:26:19 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/14 16:28:30 16:26:18 WARN Pod/odh-model-controller-6849fc49d8-ndthx: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/14 16:28:30 16:26:18 INFO Pod/odh-model-controller-6849fc49d8-ndthx: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-ndthx to ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/14 16:28:30 16:26:18 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/14 16:28:30 16:26:18 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-ndthx 2026/07/14 16:28:30 16:26:11 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-hs4lj: AddedInterface - Add eth0 [10.131.0.124/23] from ovn-kubernetes 2026/07/14 16:28:30 16:26:11 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-hs4lj: Pulled - Container image "registry.redhat.io/rhoai/odh-trainer-rhel9@sha256:59ea523e2df9c1e705c755308ac11420682081cc63d54890e2b15fbd7d8ba570" already present on machine 2026/07/14 16:28:30 16:26:11 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-hs4lj: Created - Created container: manager 2026/07/14 16:28:30 16:26:11 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-hs4lj: Started - Started container manager 2026/07/14 16:28:30 16:26:10 INFO ReplicaSet/kubeflow-trainer-controller-manager-7ccfdd66: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-7ccfdd66-hs4lj 2026/07/14 16:28:30 16:26:10 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-7ccfdd66 from 0 to 1 2026/07/14 16:28:30 16:26:10 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-hs4lj: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-7ccfdd66-hs4lj to ods-qe-psi-23-v5877-worker-0-gdv9r 2026/07/14 16:28:30 16:26:07 INFO Pod/kserve-controller-manager-6c67dcb69c-d8xbt: Killing - Stopping container manager 2026/07/14 16:28:30 16:26:07 INFO Pod/model-serving-api-6688cd56bd-2nb4x: Killing - Stopping container server 2026/07/14 16:28:30 16:26:07 INFO Pod/odh-model-controller-6849fc49d8-jxnzj: Killing - Stopping container manager 2026/07/14 16:28:30 16:26:07 INFO Pod/llmisvc-controller-manager-5d65df97bb-m8lzn: Killing - Stopping container manager 2026/07/14 16:28:30 16:24:59 INFO Pod/maas-controller-85bdcf9988-hmmld: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 1.269s (1.269s including waiting). Image size: 212866077 bytes. 2026/07/14 16:28:30 16:24:58 INFO Pod/maas-controller-85bdcf9988-hmmld: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/14 16:28:30 16:24:06 WARN Pod/maas-controller-85bdcf9988-hmmld: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-njx76); attempting to pull the image may not succeed. 2026/07/14 16:28:30 16:24:03 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/14 16:28:30 === RESOURCE QUOTAS === 2026/07/14 16:28:30 No resource quotas found in monitored namespaces 2026/07/14 16:28:30 Classification for components: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/14 16:28:30 Evidence: container maas-controller-85bdcf9988-hmmld/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-hmmld_redhat-ods-applications(eb60e58f-4f60-40cf-9cbd-1c5958018d9b) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-85bdcf9988-hmmld/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-hmmld_redhat-ods-applications(eb60e58f-4f60-40cf-9cbd-1c5958018d9b)"],"confidence":"medium"} 2026/07/14 16:28:30 Diagnostics complete for failed test: components controller_test.go:484: Stopping: components test failed. 2026/07/14 16:28:30 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/14 16:28:30 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/14 16:28:32 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (1513.36s)