=== RUN TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable helper_test.go:150: Verifying Tenant models-as-a-service/default-tenant is present before disable helper_test.go:150: Disabling MaaS subcomponent (setting to Removed) helper_test.go:150: Waiting for Tenant models-as-a-service/default-tenant to be deleted testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable (430.20s) === RUN TestOdhOperator/components/group_4/modelsasservice === PAUSE TestOdhOperator/components/group_4/modelsasservice === CONT TestOdhOperator/components/group_4/modelsasservice 2026-07-20T09:38:45Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.21.19"}} controller_test.go:753: Creating MaaS PostgreSQL instance in namespace redhat-ods-applications controller_test.go:753: MaaS PostgreSQL instance and maas-db-config secret created in namespace redhat-ods-applications controller_test.go:753: Creating MaaS GatewayClass and Gateway: openshift-ingress/maas-default-gateway controller_test.go:753: Using hostname=maas.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com, certSecret=2r2qdb90ltvhjc43lp6iqc6aqb7bgjda-primary-cert-bundle-secret controller_test.go:753: MaaS Gateway openshift-ingress/maas-default-gateway created with HTTP+HTTPS listeners test_context_test.go:615: Timed out after 420.009s. The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1824 with: Expected resource list 'models-as-a-service/default-tenant' of kind 'Tenant' to be empty, but it contains resources. Expected <[]unstructured.Unstructured | len:1, cap:1>: [ { Object: { "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "kind": <string>"Tenant", "metadata": <map[string]interface {} | len:8>{ "creationTimestamp": <string>"2026-07-20T09:39:46Z", "generation": <int64>1, "managedFields": <[]interface {} | len:2, cap:2>[ <map[string]interface {} | len:6>{ "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:2>{ "f:metadata": <map[string]interface {} | len:1>{ "f:ownerReferences": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "k:{\"uid\":\"e39d4dce-f802-448e-a6fa-02babc09eba5\"}": <map[string]interface {} | len:0>{}, }, }, "f:spec": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "f:gatewayRef": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:name": <map[string]interface {} | len:0>{}, "f:namespace": <map[string]interface {} | len:0>{}, }, }, }, "manager": <string>"manager", "operation": <string>"Update", "time": <string>"2026-07-20T09:39:46Z", "apiVersion": <string>"maas.opendatahub.io/v1alpha1", }, <map[string]interface {} | len:7>{ "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:1>{ "f:status": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:conditions": <map[string]interface {} | len:0>{}, "f:phase": <map[string]interface {} | len:0>{}, }, }, "manager": <string>"manager", "operation": <string>"Update", "subresource": <string>"status", "time": <string>"2026-07-20T09:40:11Z", "apiVersion": <string>"maas.opendatahub.io/v1alpha1", }, ], "name": <string>"default-tenant", "namespace": <string>"models-as-a-service", "ownerReferences": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:4>{ "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "kind": <string>"Config", "name": <string>"default", "uid": <string>"e39d4dce-f802-448e-a6fa-02babc09eba5", }, ], "resourceVersion": <string>"40868755", "uid": <string>"e5d39371-1d62-4588-aa1e-211e3fec8167", }, "spec": <map[string]interface {} | len:1>{ "gatewayRef": <map[string]interface {} | len:2>{ "name": <string>"maas-default-gateway", "namespace": <string>"openshift-ingress", }, }, "status": <map[string]interface {} | len:2>{ "conditions": <[]interface {} | len:5, cap:8>[ <map[string]interface {} | len:6>{ "observedGeneration": <int64>1, "reason": <string>"DependenciesMet", "status": <string>"True", "type": <string>"DependenciesAvailable", "lastTransitionTime": <string>"2026-07-20T09:39:49Z", "message": <string>"AuthConfig CRD (Kuadrant) is available on the cluster", }, <map[string]interface {} | len:6>{ "observedGeneration": <int64>1, "reason": <string>"PrerequisitesMet", "status": <string>"True", "type": <string>"MaaSPrerequisitesAvailable", "lastTransitionTime": <string>"2026-07-20T09:39:49Z", "message": <string>"All prerequisites are satisfied", }, <map[string]interface {} | len:6>{ "observedGeneration": <int64>1, "reason": <string>"PrerequisitesMet", "status": <string>"False", "type": <string>"Degraded", "lastTransitionTime": <string>"2026-07-20T09:39:49Z", "message": <string>"All prerequisites are satisfied", }, <map[string]interface {} | len:6>{ "status": <string>"True", "type": <string>"DeploymentsAvailable", "lastTransitionTime": <string>"2026-07-20T09:40:11Z", "message": <string>"maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"DeploymentsReady", }, <map[string]interface {} | len:6>{ "type": <string>"Ready", "lastTransitionTime": <string>"2026-07-20T09:40:11Z", "message": <string>"MaaS platform manifests applied and maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"Reconciled", "status": <string>"True", }, ], "phase": <string>"Active", }, }, }, ] to be empty --- FAIL: TestOdhOperator/components/group_4/modelsasservice (521.63s) === RUN TestOdhOperator/components/group_4 --- FAIL: TestOdhOperator/components/group_4 (0.00s) === RUN TestOdhOperator/components 2026/07/20 09:47:27 TEST FAILURE DETECTED: group 4 2026/07/20 09:47:27 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 4) === 2026/07/20 09:47:27 === CLUSTER STATE === 2026/07/20 09:47:27 Node: ip-10-0-1-227.us-east-2.compute.internal 2026/07/20 09:47:27 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/20 09:47:27 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/20 09:47:27 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/20 09:47:27 Condition Ready: True - kubelet is posting ready status 2026/07/20 09:47:27 Resources: 15500m CPU, 63871016Ki memory allocatable (of 16000m CPU, 65021992Ki memory capacity) 2026/07/20 09:47:27 Node: ip-10-0-1-23.us-east-2.compute.internal 2026/07/20 09:47:27 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/20 09:47:27 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/20 09:47:27 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/20 09:47:27 Condition Ready: True - kubelet is posting ready status 2026/07/20 09:47:27 Resources: 15500m CPU, 63182900Ki memory allocatable (of 16000m CPU, 64333876Ki memory capacity) 2026/07/20 09:47:27 Node: ip-10-0-1-45.us-east-2.compute.internal 2026/07/20 09:47:27 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/20 09:47:27 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/20 09:47:27 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/20 09:47:27 Condition Ready: True - kubelet is posting ready status 2026/07/20 09:47:27 Resources: 15500m CPU, 63871012Ki memory allocatable (of 16000m CPU, 65021988Ki memory capacity) 2026/07/20 09:47:27 === DEPLOYMENTS === 2026/07/20 09:47:27 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/20 09:47:27 === PODS === 2026/07/20 09:47:27 Failed to collect pod data: redhat-ods-applications/maas-controller-75bbfd978-tc47b: container manager not ready 2026/07/20 09:47:27 === OPERATOR STATUS === 2026/07/20 09:47:27 Operator deployment: 3/3 ready 2026/07/20 09:47:27 === DSCI STATUS === 2026/07/20 09:47:27 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/20 09:47:27 === DSC STATUS === 2026/07/20 09:47:27 DSC default-dsc: 2026/07/20 09:47:27 Ready: True - 2026/07/20 09:47:27 ProvisioningSucceeded: True - 2026/07/20 09:47:27 ComponentsReady: True - 2026/07/20 09:47:27 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/20 09:47:27 DashboardReady: False - Component ManagementState is set to Removed 2026/07/20 09:47:27 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/20 09:47:27 KserveLLMInferenceServiceDependencies: True - 2026/07/20 09:47:27 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/20 09:47:27 KserveReady: True - 2026/07/20 09:47:27 KueueReady: False - Component ManagementState is set to Removed 2026/07/20 09:47:27 LLM-D-WVADependencies: True - 2026/07/20 09:47:27 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/20 09:47:27 ModelControllerReady: True - 2026/07/20 09:47:27 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/20 09:47:27 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/20 09:47:27 OGXReady: False - Component ManagementState is set to Removed 2026/07/20 09:47:27 RayReady: False - Component ManagementState is set to Removed 2026/07/20 09:47:27 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/20 09:47:27 TrainerReady: False - Component ManagementState is set to Removed 2026/07/20 09:47:27 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/20 09:47:27 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/20 09:47:27 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/20 09:47:27 === RECENT EVENTS === 2026/07/20 09:47:27 09:46:35 INFO Pod/maas-controller-75bbfd978-tc47b: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/20 09:47:27 09:46:35 INFO Pod/maas-controller-75bbfd978-tc47b: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 543ms (543ms including waiting). Image size: 212866077 bytes. 2026/07/20 09:47:27 09:45:31 WARN Pod/maas-controller-75bbfd978-tc47b: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-kd4ck); attempting to pull the image may not succeed. 2026/07/20 09:47:27 09:45:07 INFO Job/maas-api-key-cleanup-29742345: Completed - Job completed 2026/07/20 09:47:27 09:45:07 INFO CronJob/maas-api-key-cleanup: SawCompletedJob - Saw completed job: maas-api-key-cleanup-29742345, condition: Complete 2026/07/20 09:47:27 09:45:01 INFO Pod/maas-api-key-cleanup-29742345-qdl6h: Started - Started container cleanup 2026/07/20 09:47:27 09:45:01 INFO Pod/maas-api-key-cleanup-29742345-qdl6h: Created - Created container: cleanup 2026/07/20 09:47:27 09:45:01 INFO Pod/maas-api-key-cleanup-29742345-qdl6h: Pulled - Container image "registry.redhat.io/ubi9/ubi-minimal@sha256:907b68736aa798b2d38255b7aa070b2a70acb90803864a40f05d0ec47556ddd0" already present on machine 2026/07/20 09:47:27 09:45:01 INFO Pod/maas-api-key-cleanup-29742345-qdl6h: AddedInterface - Add eth0 [10.130.0.146/23] from ovn-kubernetes 2026/07/20 09:47:27 09:45:00 INFO Pod/maas-api-key-cleanup-29742345-qdl6h: Scheduled - Successfully assigned redhat-ods-applications/maas-api-key-cleanup-29742345-qdl6h to ip-10-0-1-23.us-east-2.compute.internal 2026/07/20 09:47:27 09:45:00 INFO Job/maas-api-key-cleanup-29742345: SuccessfulCreate - Created pod: maas-api-key-cleanup-29742345-qdl6h 2026/07/20 09:47:27 09:45:00 INFO CronJob/maas-api-key-cleanup: SuccessfulCreate - Created job maas-api-key-cleanup-29742345 2026/07/20 09:47:27 09:43:48 INFO Pod/maas-controller-75bbfd978-tc47b: Created - Created container: manager 2026/07/20 09:47:27 09:43:48 INFO Pod/maas-controller-75bbfd978-tc47b: Started - Started container manager 2026/07/20 09:47:27 09:43:48 INFO Pod/maas-controller-75bbfd978-tc47b: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 657ms (657ms including waiting). Image size: 212866077 bytes. 2026/07/20 09:47:27 === RESOURCE QUOTAS === 2026/07/20 09:47:27 No resource quotas found in monitored namespaces 2026/07/20 09:47:27 Classification for group 4: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/20 09:47:27 Evidence: container maas-controller-75bbfd978-tc47b/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-75bbfd978-tc47b_redhat-ods-applications(b21a9762-2336-4c7c-b618-ad5dadb75036) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-75bbfd978-tc47b/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-75bbfd978-tc47b_redhat-ods-applications(b21a9762-2336-4c7c-b618-ad5dadb75036)"],"confidence":"medium"} 2026/07/20 09:47:27 Diagnostics complete for failed test: group 4 controller_test.go:753: Stopping: group 4 test failed. 2026/07/20 09:47:27 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/20 09:47:27 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/20 09:48:38 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (1464.24s) === RUN TestOdhOperator 2026/07/20 09:25:04 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/20 09:49:39 TEST FAILURE DETECTED: components 2026/07/20 09:49:39 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/20 09:49:39 === CLUSTER STATE === 2026/07/20 09:49:39 Node: ip-10-0-1-227.us-east-2.compute.internal 2026/07/20 09:49:39 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/20 09:49:39 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/20 09:49:39 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/20 09:49:39 Condition Ready: True - kubelet is posting ready status 2026/07/20 09:49:39 Resources: 15500m CPU, 63871016Ki memory allocatable (of 16000m CPU, 65021992Ki memory capacity) 2026/07/20 09:49:39 Node: ip-10-0-1-23.us-east-2.compute.internal 2026/07/20 09:49:39 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/20 09:49:39 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/20 09:49:39 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/20 09:49:39 Condition Ready: True - kubelet is posting ready status 2026/07/20 09:49:39 Resources: 15500m CPU, 63182900Ki memory allocatable (of 16000m CPU, 64333876Ki memory capacity) 2026/07/20 09:49:39 Node: ip-10-0-1-45.us-east-2.compute.internal 2026/07/20 09:49:39 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/20 09:49:39 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/20 09:49:39 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/20 09:49:39 Condition Ready: True - kubelet is posting ready status 2026/07/20 09:49:39 Resources: 15500m CPU, 63871012Ki memory allocatable (of 16000m CPU, 65021988Ki memory capacity) 2026/07/20 09:49:39 === DEPLOYMENTS === 2026/07/20 09:49:39 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/20 09:49:39 === PODS === 2026/07/20 09:49:39 Failed to collect pod data: redhat-ods-applications/maas-controller-75bbfd978-tc47b: container manager not ready 2026/07/20 09:49:39 === OPERATOR STATUS === 2026/07/20 09:49:39 Operator deployment: 3/3 ready 2026/07/20 09:49:39 === DSCI STATUS === 2026/07/20 09:49:39 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/20 09:49:39 === DSC STATUS === 2026/07/20 09:49:39 DSC default-dsc: 2026/07/20 09:49:39 Ready: True - 2026/07/20 09:49:39 ProvisioningSucceeded: True - 2026/07/20 09:49:39 ComponentsReady: True - 2026/07/20 09:49:39 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/20 09:49:39 DashboardReady: False - Component ManagementState is set to Removed 2026/07/20 09:49:39 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/20 09:49:39 KserveLLMInferenceServiceDependencies: True - 2026/07/20 09:49:39 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/20 09:49:39 KserveReady: True - 2026/07/20 09:49:39 KueueReady: False - Component ManagementState is set to Removed 2026/07/20 09:49:39 LLM-D-WVADependencies: True - 2026/07/20 09:49:39 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/20 09:49:39 ModelControllerReady: True - 2026/07/20 09:49:39 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/20 09:49:39 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/20 09:49:39 OGXReady: False - Component ManagementState is set to Removed 2026/07/20 09:49:39 RayReady: False - Component ManagementState is set to Removed 2026/07/20 09:49:39 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/20 09:49:39 TrainerReady: True - 2026/07/20 09:49:39 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/20 09:49:39 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/20 09:49:39 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/20 09:49:39 === RECENT EVENTS === 2026/07/20 09:49:39 09:49:22 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-7b46b9fbbb-w2k7h_fd99a930-9bfa-4b51-907b-a14f6b4dabdf became leader 2026/07/20 09:49:39 09:49:21 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-dd9f6cf9-xzlws_dee59c2c-6334-439c-bb2a-65112973a17f became leader 2026/07/20 09:49:39 09:49:09 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-bd4c5cbcb-znz65_d073fa90-0fe8-44c8-b59f-1bafd5a7a5d4 became leader 2026/07/20 09:49:39 09:49:04 INFO Pod/kserve-controller-manager-dd9f6cf9-xzlws: Created - Created container: manager 2026/07/20 09:49:39 09:49:04 INFO Pod/kserve-controller-manager-dd9f6cf9-xzlws: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 536ms (536ms including waiting). Image size: 238435613 bytes. 2026/07/20 09:49:39 09:49:04 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-w2k7h: Started - Started container manager 2026/07/20 09:49:39 09:49:04 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-w2k7h: Created - Created container: manager 2026/07/20 09:49:39 09:49:04 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-w2k7h: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 461ms (461ms including waiting). Image size: 236366599 bytes. 2026/07/20 09:49:39 09:49:04 INFO Pod/kserve-controller-manager-dd9f6cf9-xzlws: Started - Started container manager 2026/07/20 09:49:39 09:49:03 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-w2k7h: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-w2k7h to ip-10-0-1-45.us-east-2.compute.internal 2026/07/20 09:49:39 09:49:03 INFO Pod/kserve-controller-manager-dd9f6cf9-xzlws: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-dd9f6cf9-xzlws to ip-10-0-1-23.us-east-2.compute.internal 2026/07/20 09:49:39 09:49:03 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-w2k7h: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/20 09:49:39 09:49:03 INFO Pod/kserve-controller-manager-dd9f6cf9-xzlws: AddedInterface - Add eth0 [10.130.0.154/23] from ovn-kubernetes 2026/07/20 09:49:39 09:49:03 INFO Pod/kserve-controller-manager-dd9f6cf9-xzlws: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/20 09:49:39 09:49:03 INFO ReplicaSet/llmisvc-controller-manager-7b46b9fbbb: SuccessfulCreate - Created pod: llmisvc-controller-manager-7b46b9fbbb-w2k7h 2026/07/20 09:49:39 09:49:03 INFO ReplicaSet/kserve-controller-manager-dd9f6cf9: SuccessfulCreate - Created pod: kserve-controller-manager-dd9f6cf9-xzlws 2026/07/20 09:49:39 09:49:03 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-w2k7h: AddedInterface - Add eth0 [10.129.0.252/23] from ovn-kubernetes 2026/07/20 09:49:39 09:49:03 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-7b46b9fbbb from 0 to 1 2026/07/20 09:49:39 09:49:03 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-dd9f6cf9 from 0 to 1 2026/07/20 09:49:39 09:48:52 INFO Pod/model-serving-api-5679d6b4f-pvgww: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/20 09:49:39 09:48:52 INFO Pod/model-serving-api-5679d6b4f-pvgww: AddedInterface - Add eth0 [10.130.0.153/23] from ovn-kubernetes 2026/07/20 09:49:39 09:48:52 INFO Pod/model-serving-api-5679d6b4f-pvgww: Created - Created container: server 2026/07/20 09:49:39 09:48:52 INFO Pod/model-serving-api-5679d6b4f-pvgww: Started - Started container server 2026/07/20 09:49:39 09:48:51 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/20 09:49:39 09:48:51 INFO Pod/model-serving-api-5679d6b4f-pvgww: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5679d6b4f-pvgww to ip-10-0-1-23.us-east-2.compute.internal 2026/07/20 09:49:39 09:48:51 WARN ReplicaSet/model-serving-api-5679d6b4f: FailedCreate - Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/20 09:49:39 09:48:51 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5679d6b4f from 0 to 1 2026/07/20 09:49:39 09:48:51 INFO Pod/odh-model-controller-bd4c5cbcb-znz65: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/20 09:49:39 09:48:51 INFO Pod/odh-model-controller-bd4c5cbcb-znz65: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 453ms (453ms including waiting). Image size: 238959091 bytes. 2026/07/20 09:49:39 09:48:51 INFO Pod/odh-model-controller-bd4c5cbcb-znz65: Created - Created container: manager 2026/07/20 09:49:39 09:48:51 INFO Pod/odh-model-controller-bd4c5cbcb-znz65: Started - Started container manager 2026/07/20 09:49:39 09:48:51 INFO ReplicaSet/model-serving-api-5679d6b4f: SuccessfulCreate - Created pod: model-serving-api-5679d6b4f-pvgww 2026/07/20 09:49:39 09:48:50 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/20 09:49:39 09:48:50 WARN Pod/odh-model-controller-bd4c5cbcb-znz65: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/20 09:49:39 09:48:50 INFO Pod/odh-model-controller-bd4c5cbcb-znz65: AddedInterface - Add eth0 [10.130.0.152/23] from ovn-kubernetes 2026/07/20 09:49:39 09:48:49 INFO Pod/odh-model-controller-bd4c5cbcb-znz65: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-bd4c5cbcb-znz65 to ip-10-0-1-23.us-east-2.compute.internal 2026/07/20 09:49:39 09:48:49 INFO ReplicaSet/odh-model-controller-bd4c5cbcb: SuccessfulCreate - Created pod: odh-model-controller-bd4c5cbcb-znz65 2026/07/20 09:49:39 09:48:49 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-bd4c5cbcb from 0 to 1 2026/07/20 09:49:39 09:48:40 INFO Pod/odh-model-controller-bd4c5cbcb-5nctv: Killing - Stopping container manager 2026/07/20 09:49:39 09:48:39 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/20 09:49:39 09:48:39 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-jmk2z: Killing - Stopping container manager 2026/07/20 09:49:39 09:48:39 INFO Pod/model-serving-api-5679d6b4f-j466g: Killing - Stopping container server 2026/07/20 09:49:39 09:48:39 INFO Pod/kserve-controller-manager-dd9f6cf9-vv4dz: Killing - Stopping container manager 2026/07/20 09:49:39 09:48:16 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-7b46b9fbbb-jmk2z_c2d2216c-c01d-432c-9dc0-f997b16306f3 became leader 2026/07/20 09:49:39 09:48:15 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-dd9f6cf9-vv4dz_1c1e34e4-afd7-4efc-bb3d-64d6fc9bd981 became leader 2026/07/20 09:49:39 09:47:59 INFO Pod/kserve-controller-manager-dd9f6cf9-vv4dz: Started - Started container manager 2026/07/20 09:49:39 09:47:59 INFO Pod/kserve-controller-manager-dd9f6cf9-vv4dz: Created - Created container: manager 2026/07/20 09:49:39 09:47:59 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-jmk2z: Started - Started container manager 2026/07/20 09:49:39 09:47:59 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-jmk2z: Created - Created container: manager 2026/07/20 09:49:39 09:47:59 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-jmk2z: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 502ms (502ms including waiting). Image size: 236366599 bytes. 2026/07/20 09:49:39 09:47:59 INFO Pod/kserve-controller-manager-dd9f6cf9-vv4dz: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 567ms (567ms including waiting). Image size: 238435613 bytes. 2026/07/20 09:49:39 09:47:58 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-jmk2z: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-jmk2z to ip-10-0-1-23.us-east-2.compute.internal 2026/07/20 09:49:39 09:47:58 INFO Pod/kserve-controller-manager-dd9f6cf9-vv4dz: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-dd9f6cf9-vv4dz to ip-10-0-1-23.us-east-2.compute.internal 2026/07/20 09:49:39 09:47:58 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-7b46b9fbbb from 0 to 1 2026/07/20 09:49:39 09:47:58 INFO ReplicaSet/kserve-controller-manager-dd9f6cf9: SuccessfulCreate - Created pod: kserve-controller-manager-dd9f6cf9-vv4dz 2026/07/20 09:49:39 09:47:58 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-jmk2z: AddedInterface - Add eth0 [10.130.0.151/23] from ovn-kubernetes 2026/07/20 09:49:39 09:47:58 INFO ReplicaSet/llmisvc-controller-manager-7b46b9fbbb: SuccessfulCreate - Created pod: llmisvc-controller-manager-7b46b9fbbb-jmk2z 2026/07/20 09:49:39 09:47:58 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-jmk2z: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/20 09:49:39 09:47:58 INFO Pod/kserve-controller-manager-dd9f6cf9-vv4dz: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/20 09:49:39 09:47:58 INFO Pod/kserve-controller-manager-dd9f6cf9-vv4dz: AddedInterface - Add eth0 [10.130.0.150/23] from ovn-kubernetes 2026/07/20 09:49:39 09:47:57 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-dd9f6cf9 from 0 to 1 2026/07/20 09:49:39 09:47:56 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-bd4c5cbcb-5nctv_29f80d58-249f-4d0a-b783-e4c774c68133 became leader 2026/07/20 09:49:39 09:47:49 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-b5fd84b9c-kbcqr_5433eb95-c21c-42c5-870c-97f0188f863e became leader 2026/07/20 09:49:39 09:47:42 INFO Pod/model-serving-api-5679d6b4f-j466g: Started - Started container server 2026/07/20 09:49:39 09:47:42 INFO Pod/model-serving-api-5679d6b4f-j466g: Created - Created container: server 2026/07/20 09:49:39 09:47:41 INFO Pod/odh-model-controller-bd4c5cbcb-5nctv: Started - Started container manager 2026/07/20 09:49:39 09:47:41 INFO Pod/model-serving-api-5679d6b4f-j466g: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/20 09:49:39 09:47:41 INFO Pod/odh-model-controller-bd4c5cbcb-5nctv: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 494ms (494ms including waiting). Image size: 238959091 bytes. 2026/07/20 09:49:39 09:47:41 INFO Pod/odh-model-controller-bd4c5cbcb-5nctv: Created - Created container: manager 2026/07/20 09:49:39 09:47:41 INFO Pod/model-serving-api-5679d6b4f-j466g: AddedInterface - Add eth0 [10.130.0.149/23] from ovn-kubernetes 2026/07/20 09:49:39 09:47:40 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/20 09:49:39 09:47:40 INFO Pod/model-serving-api-5679d6b4f-j466g: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5679d6b4f-j466g to ip-10-0-1-23.us-east-2.compute.internal 2026/07/20 09:49:39 09:47:40 WARN Pod/model-serving-api-5679d6b4f-j466g: FailedMount - MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 2026/07/20 09:49:39 09:47:40 WARN ReplicaSet/model-serving-api-5679d6b4f: FailedCreate - Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/20 09:49:39 09:47:40 INFO ReplicaSet/model-serving-api-5679d6b4f: SuccessfulCreate - Created pod: model-serving-api-5679d6b4f-j466g 2026/07/20 09:49:39 09:47:40 INFO Pod/odh-model-controller-bd4c5cbcb-5nctv: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/20 09:49:39 09:47:40 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5679d6b4f from 0 to 1 2026/07/20 09:49:39 09:47:40 INFO Pod/odh-model-controller-bd4c5cbcb-5nctv: AddedInterface - Add eth0 [10.130.0.148/23] from ovn-kubernetes 2026/07/20 09:49:39 09:47:39 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/20 09:49:39 09:47:39 INFO Pod/odh-model-controller-bd4c5cbcb-5nctv: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-bd4c5cbcb-5nctv to ip-10-0-1-23.us-east-2.compute.internal 2026/07/20 09:49:39 09:47:39 WARN Pod/odh-model-controller-bd4c5cbcb-5nctv: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/20 09:49:39 09:47:39 INFO ReplicaSet/odh-model-controller-bd4c5cbcb: SuccessfulCreate - Created pod: odh-model-controller-bd4c5cbcb-5nctv 2026/07/20 09:49:39 09:47:39 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-bd4c5cbcb from 0 to 1 2026/07/20 09:49:39 09:47:32 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-kbcqr: AddedInterface - Add eth0 [10.131.0.245/23] from ovn-kubernetes 2026/07/20 09:49:39 09:47:32 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-kbcqr: Started - Started container manager 2026/07/20 09:49:39 09:47:32 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-kbcqr: Created - Created container: manager 2026/07/20 09:49:39 09:47:32 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-kbcqr: Pulled - Container image "quay.io/rhoai/odh-trainer-rhel9@sha256:59ea523e2df9c1e705c755308ac11420682081cc63d54890e2b15fbd7d8ba570" already present on machine 2026/07/20 09:49:39 09:47:31 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-kbcqr: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-b5fd84b9c-kbcqr to ip-10-0-1-227.us-east-2.compute.internal 2026/07/20 09:49:39 09:47:31 INFO ReplicaSet/kubeflow-trainer-controller-manager-b5fd84b9c: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-b5fd84b9c-kbcqr 2026/07/20 09:49:39 09:47:31 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-b5fd84b9c from 0 to 1 2026/07/20 09:49:39 09:47:29 INFO Pod/model-serving-api-5679d6b4f-r8kqr: Killing - Stopping container server 2026/07/20 09:49:39 09:47:29 INFO Pod/odh-model-controller-bd4c5cbcb-4fjc2: Killing - Stopping container manager 2026/07/20 09:49:39 09:47:28 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/20 09:49:39 09:47:28 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-ff6wc: Killing - Stopping container manager 2026/07/20 09:49:39 09:47:28 INFO Pod/kserve-controller-manager-dd9f6cf9-9dl57: Killing - Stopping container manager 2026/07/20 09:49:39 09:47:28 WARN Endpoints/llmisvc-controller-manager-service: FailedToUpdateEndpoint - Failed to update endpoint redhat-ods-applications/llmisvc-controller-manager-service: Operation cannot be fulfilled on endpoints "llmisvc-controller-manager-service": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/llmisvc-controller-manager-service, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 30bb22c0-823b-4687-8a3e-77e2a519cb98, UID in object meta: 2026/07/20 09:49:39 09:46:35 INFO Pod/maas-controller-75bbfd978-tc47b: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/20 09:49:39 09:46:35 INFO Pod/maas-controller-75bbfd978-tc47b: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 543ms (543ms including waiting). Image size: 212866077 bytes. 2026/07/20 09:49:39 09:45:31 WARN Pod/maas-controller-75bbfd978-tc47b: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-kd4ck); attempting to pull the image may not succeed. 2026/07/20 09:49:39 09:45:07 INFO CronJob/maas-api-key-cleanup: SawCompletedJob - Saw completed job: maas-api-key-cleanup-29742345, condition: Complete 2026/07/20 09:49:39 09:45:07 INFO Job/maas-api-key-cleanup-29742345: Completed - Job completed 2026/07/20 09:49:39 09:45:01 INFO Pod/maas-api-key-cleanup-29742345-qdl6h: AddedInterface - Add eth0 [10.130.0.146/23] from ovn-kubernetes 2026/07/20 09:49:39 09:45:01 INFO Pod/maas-api-key-cleanup-29742345-qdl6h: Pulled - Container image "registry.redhat.io/ubi9/ubi-minimal@sha256:907b68736aa798b2d38255b7aa070b2a70acb90803864a40f05d0ec47556ddd0" already present on machine 2026/07/20 09:49:39 09:45:01 INFO Pod/maas-api-key-cleanup-29742345-qdl6h: Created - Created container: cleanup 2026/07/20 09:49:39 09:45:01 INFO Pod/maas-api-key-cleanup-29742345-qdl6h: Started - Started container cleanup 2026/07/20 09:49:39 09:45:00 INFO Pod/maas-api-key-cleanup-29742345-qdl6h: Scheduled - Successfully assigned redhat-ods-applications/maas-api-key-cleanup-29742345-qdl6h to ip-10-0-1-23.us-east-2.compute.internal 2026/07/20 09:49:39 09:45:00 INFO Job/maas-api-key-cleanup-29742345: SuccessfulCreate - Created pod: maas-api-key-cleanup-29742345-qdl6h 2026/07/20 09:49:39 09:45:00 INFO CronJob/maas-api-key-cleanup: SuccessfulCreate - Created job maas-api-key-cleanup-29742345 2026/07/20 09:49:39 === RESOURCE QUOTAS === 2026/07/20 09:49:39 No resource quotas found in monitored namespaces 2026/07/20 09:49:39 Classification for components: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/20 09:49:39 Evidence: container maas-controller-75bbfd978-tc47b/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-75bbfd978-tc47b_redhat-ods-applications(b21a9762-2336-4c7c-b618-ad5dadb75036) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-75bbfd978-tc47b/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-75bbfd978-tc47b_redhat-ods-applications(b21a9762-2336-4c7c-b618-ad5dadb75036)"],"confidence":"medium"} 2026/07/20 09:49:39 Diagnostics complete for failed test: components controller_test.go:484: Stopping: components test failed. 2026/07/20 09:49:39 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/20 09:49:39 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/20 09:49:40 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (1476.75s)