=== RUN TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable helper_test.go:150: Verifying Tenant models-as-a-service/default-tenant is present before disable helper_test.go:150: Disabling MaaS subcomponent (setting to Removed) helper_test.go:150: Waiting for Tenant models-as-a-service/default-tenant to be deleted testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable (430.26s) === RUN TestOdhOperator/components/group_4/modelsasservice === PAUSE TestOdhOperator/components/group_4/modelsasservice === CONT TestOdhOperator/components/group_4/modelsasservice 2026-07-12T11:01:22Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.20.12"}} controller_test.go:753: Creating MaaS PostgreSQL instance in namespace redhat-ods-applications controller_test.go:753: MaaS PostgreSQL instance and maas-db-config secret created in namespace redhat-ods-applications controller_test.go:753: Creating MaaS GatewayClass and Gateway: openshift-ingress/maas-default-gateway controller_test.go:753: Using hostname=maas.apps.rosa.rh-nightly-pm.nuth.p3.openshiftapps.com, certSecret=2o8clk8b6cj832lmkd2oj78crnf7qn6d-primary-cert-bundle-secret controller_test.go:753: MaaS Gateway openshift-ingress/maas-default-gateway created with HTTP+HTTPS listeners test_context_test.go:615: Timed out after 420.007s. The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1824 with: Expected resource list 'models-as-a-service/default-tenant' of kind 'Tenant' to be empty, but it contains resources. Expected <[]unstructured.Unstructured | len:1, cap:1>: [ { Object: { "spec": <map[string]interface {} | len:1>{ "gatewayRef": <map[string]interface {} | len:2>{ "name": <string>"maas-default-gateway", "namespace": <string>"openshift-ingress", }, }, "status": <map[string]interface {} | len:2>{ "conditions": <[]interface {} | len:5, cap:8>[ <map[string]interface {} | len:6>{ "status": <string>"True", "type": <string>"DependenciesAvailable", "lastTransitionTime": <string>"2026-07-12T11:02:27Z", "message": <string>"AuthConfig CRD (Kuadrant) is available on the cluster", "observedGeneration": <int64>1, "reason": <string>"DependenciesMet", }, <map[string]interface {} | len:6>{ "reason": <string>"PrerequisitesMet", "status": <string>"True", "type": <string>"MaaSPrerequisitesAvailable", "lastTransitionTime": <string>"2026-07-12T11:02:27Z", "message": <string>"All prerequisites are satisfied", "observedGeneration": <int64>1, }, <map[string]interface {} | len:6>{ "message": <string>"All prerequisites are satisfied", "observedGeneration": <int64>1, "reason": <string>"PrerequisitesMet", "status": <string>"False", "type": <string>"Degraded", "lastTransitionTime": <string>"2026-07-12T11:02:27Z", }, <map[string]interface {} | len:6>{ "message": <string>"maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"DeploymentsReady", "status": <string>"True", "type": <string>"DeploymentsAvailable", "lastTransitionTime": <string>"2026-07-12T11:02:49Z", }, <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-07-12T11:02:49Z", "message": <string>"MaaS platform manifests applied and maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"Reconciled", "status": <string>"True", "type": <string>"Ready", }, ], "phase": <string>"Active", }, "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "kind": <string>"Tenant", "metadata": <map[string]interface {} | len:8>{ "resourceVersion": <string>"212372294", "uid": <string>"0f48ecc9-08ce-430b-8cee-943c35093179", "creationTimestamp": <string>"2026-07-12T11:02:24Z", "generation": <int64>1, "managedFields": <[]interface {} | len:2, cap:2>[ <map[string]interface {} | len:6>{ "fieldsV1": <map[string]interface {} | len:2>{ "f:metadata": <map[string]interface {} | len:1>{ "f:ownerReferences": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "k:{\"uid\":\"095df431-95a9-41ec-92f0-3f1d28130405\"}": <map[string]interface {} | len:0>{}, }, }, "f:spec": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "f:gatewayRef": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:name": <map[string]interface {} | len:0>{}, "f:namespace": <map[string]interface {} | len:0>{}, }, }, }, "manager": <string>"manager", "operation": <string>"Update", "time": <string>"2026-07-12T11:02:24Z", "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "fieldsType": <string>"FieldsV1", }, <map[string]interface {} | len:7>{ "manager": <string>"manager", "operation": <string>"Update", "subresource": <string>"status", "time": <string>"2026-07-12T11:02:49Z", "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:1>{ "f:status": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:conditions": <map[string]interface {} | len:0>{}, "f:phase": <map[string]interface {} | len:0>{}, }, }, }, ], "name": <string>"default-tenant", "namespace": <string>"models-as-a-service", "ownerReferences": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:4>{ "uid": <string>"095df431-95a9-41ec-92f0-3f1d28130405", "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "kind": <string>"Config", "name": <string>"default", }, ], }, }, }, ] to be empty --- FAIL: TestOdhOperator/components/group_4/modelsasservice (521.82s) === RUN TestOdhOperator/components/group_4 --- FAIL: TestOdhOperator/components/group_4 (0.00s) === RUN TestOdhOperator/components 2026/07/12 11:10:04 TEST FAILURE DETECTED: group 4 2026/07/12 11:10:04 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 4) === 2026/07/12 11:10:05 === CLUSTER STATE === 2026/07/12 11:10:05 Node: ip-10-0-1-120.us-east-2.compute.internal 2026/07/12 11:10:05 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:10:05 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:10:05 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:10:05 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:10:05 Resources: 7500m CPU, 31236228Ki memory allocatable (of 8000m CPU, 32387204Ki memory capacity) 2026/07/12 11:10:05 Node: ip-10-0-1-132.us-east-2.compute.internal 2026/07/12 11:10:05 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:10:05 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:10:05 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:10:05 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:10:05 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/12 11:10:05 Node: ip-10-0-1-39.us-east-2.compute.internal 2026/07/12 11:10:05 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:10:05 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:10:05 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:10:05 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:10:05 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/12 11:10:05 === DEPLOYMENTS === 2026/07/12 11:10:05 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/12 11:10:05 === PODS === 2026/07/12 11:10:05 Failed to collect pod data: redhat-ods-applications/kserve-controller-manager-6c67dcb69c-d279d: container manager restarts=1; redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-7zsl6: container manager restarts=1; redhat-ods-applications/maas-controller-85bdcf9988-cpqkg: container manager not ready; redhat-ods-applications/odh-model-controller-6849fc49d8-mbk8w: container manager restarts=1 2026/07/12 11:10:05 === OPERATOR STATUS === 2026/07/12 11:10:05 Operator deployment: 3/3 ready 2026/07/12 11:10:05 === DSCI STATUS === 2026/07/12 11:10:05 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/12 11:10:05 === DSC STATUS === 2026/07/12 11:10:05 DSC default-dsc: 2026/07/12 11:10:05 Ready: True - 2026/07/12 11:10:05 ProvisioningSucceeded: True - 2026/07/12 11:10:05 ComponentsReady: True - 2026/07/12 11:10:05 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/12 11:10:05 DashboardReady: False - Component ManagementState is set to Removed 2026/07/12 11:10:05 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/12 11:10:05 KserveLLMInferenceServiceDependencies: True - 2026/07/12 11:10:05 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/12 11:10:05 KserveReady: True - 2026/07/12 11:10:05 KueueReady: False - Component ManagementState is set to Removed 2026/07/12 11:10:05 LLM-D-WVADependencies: True - 2026/07/12 11:10:05 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/12 11:10:05 ModelControllerReady: True - 2026/07/12 11:10:05 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/12 11:10:05 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/12 11:10:05 OGXReady: False - Component ManagementState is set to Removed 2026/07/12 11:10:05 RayReady: False - Component ManagementState is set to Removed 2026/07/12 11:10:05 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/12 11:10:05 TrainerReady: False - Component ManagementState is set to Removed 2026/07/12 11:10:05 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/12 11:10:05 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/12 11:10:05 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/12 11:10:05 === RECENT EVENTS === 2026/07/12 11:10:05 11:08:40 INFO Pod/maas-controller-85bdcf9988-cpqkg: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 525ms (525ms including waiting). Image size: 212866077 bytes. 2026/07/12 11:10:05 11:08:40 INFO Pod/maas-controller-85bdcf9988-cpqkg: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/12 11:10:05 11:07:22 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-mbk8w_c3b3df43-094f-4556-9c5f-e3337e31f8ee became leader 2026/07/12 11:10:05 11:07:21 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-d279d_7b493f4c-a57b-4908-b7e1-842eeb2c8b24 became leader 2026/07/12 11:10:05 11:07:18 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-7zsl6_6d06f0af-d85b-4971-9d0a-c82fb3fb3011 became leader 2026/07/12 11:10:05 11:07:03 INFO Pod/odh-model-controller-6849fc49d8-mbk8w: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 554ms (554ms including waiting). Image size: 238959091 bytes. 2026/07/12 11:10:05 11:07:03 INFO Pod/odh-model-controller-6849fc49d8-mbk8w: Created - Created container: manager 2026/07/12 11:10:05 11:07:03 INFO Pod/odh-model-controller-6849fc49d8-mbk8w: Started - Started container manager 2026/07/12 11:10:05 11:07:02 INFO Pod/odh-model-controller-6849fc49d8-mbk8w: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/12 11:10:05 11:07:01 INFO Pod/llmisvc-controller-manager-5d65df97bb-7zsl6: Created - Created container: manager 2026/07/12 11:10:05 11:07:01 INFO Pod/kserve-controller-manager-6c67dcb69c-d279d: Created - Created container: manager 2026/07/12 11:10:05 11:07:01 INFO Pod/llmisvc-controller-manager-5d65df97bb-7zsl6: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 556ms (556ms including waiting). Image size: 236366599 bytes. 2026/07/12 11:10:05 11:07:01 INFO Pod/llmisvc-controller-manager-5d65df97bb-7zsl6: Started - Started container manager 2026/07/12 11:10:05 11:07:01 INFO Pod/kserve-controller-manager-6c67dcb69c-d279d: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 541ms (541ms including waiting). Image size: 238435613 bytes. 2026/07/12 11:10:05 11:07:01 INFO Pod/kserve-controller-manager-6c67dcb69c-d279d: Started - Started container manager 2026/07/12 11:10:05 11:07:00 INFO Pod/llmisvc-controller-manager-5d65df97bb-7zsl6: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/12 11:10:05 11:07:00 INFO Pod/kserve-controller-manager-6c67dcb69c-d279d: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/12 11:10:05 11:06:41 WARN Pod/maas-controller-85bdcf9988-cpqkg: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-9hzjh); attempting to pull the image may not succeed. 2026/07/12 11:10:05 11:05:56 INFO Pod/maas-controller-85bdcf9988-cpqkg: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 514ms (514ms including waiting). Image size: 212866077 bytes. 2026/07/12 11:10:05 11:05:56 INFO Pod/maas-controller-85bdcf9988-cpqkg: Created - Created container: manager 2026/07/12 11:10:05 11:05:56 INFO Pod/maas-controller-85bdcf9988-cpqkg: Started - Started container manager 2026/07/12 11:10:05 === RESOURCE QUOTAS === 2026/07/12 11:10:05 No resource quotas found in monitored namespaces 2026/07/12 11:10:05 Classification for group 4: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/12 11:10:05 Evidence: container maas-controller-85bdcf9988-cpqkg/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-cpqkg_redhat-ods-applications(81519e5d-224d-4651-b3b4-dd63ce584001) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-85bdcf9988-cpqkg/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-cpqkg_redhat-ods-applications(81519e5d-224d-4651-b3b4-dd63ce584001)"],"confidence":"medium"} 2026/07/12 11:10:05 Diagnostics complete for failed test: group 4 controller_test.go:753: Stopping: group 4 test failed. 2026/07/12 11:10:05 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/12 11:10:05 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/12 11:11:06 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (1514.52s) === RUN TestOdhOperator 2026/07/12 10:47:22 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/12 11:12:47 TEST FAILURE DETECTED: components 2026/07/12 11:12:47 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/12 11:12:47 === CLUSTER STATE === 2026/07/12 11:12:47 Node: ip-10-0-1-120.us-east-2.compute.internal 2026/07/12 11:12:47 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:12:47 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:12:47 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:12:47 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:12:47 Resources: 7500m CPU, 31236228Ki memory allocatable (of 8000m CPU, 32387204Ki memory capacity) 2026/07/12 11:12:47 Node: ip-10-0-1-132.us-east-2.compute.internal 2026/07/12 11:12:47 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:12:47 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:12:47 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:12:47 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:12:47 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/12 11:12:47 Node: ip-10-0-1-39.us-east-2.compute.internal 2026/07/12 11:12:47 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:12:47 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:12:47 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:12:47 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:12:47 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/12 11:12:47 === DEPLOYMENTS === 2026/07/12 11:12:47 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/12 11:12:47 === PODS === 2026/07/12 11:12:47 Failed to collect pod data: redhat-ods-applications/maas-controller-85bdcf9988-cpqkg: container manager not ready 2026/07/12 11:12:47 === OPERATOR STATUS === 2026/07/12 11:12:47 Operator deployment: 3/3 ready 2026/07/12 11:12:47 === DSCI STATUS === 2026/07/12 11:12:47 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/12 11:12:47 === DSC STATUS === 2026/07/12 11:12:47 DSC default-dsc: 2026/07/12 11:12:47 Ready: True - 2026/07/12 11:12:47 ProvisioningSucceeded: True - 2026/07/12 11:12:47 ComponentsReady: True - 2026/07/12 11:12:47 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/12 11:12:47 DashboardReady: False - Component ManagementState is set to Removed 2026/07/12 11:12:47 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/12 11:12:47 KserveLLMInferenceServiceDependencies: True - 2026/07/12 11:12:47 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/12 11:12:47 KserveReady: True - 2026/07/12 11:12:47 KueueReady: False - Component ManagementState is set to Removed 2026/07/12 11:12:47 LLM-D-WVADependencies: True - 2026/07/12 11:12:47 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/12 11:12:47 ModelControllerReady: True - 2026/07/12 11:12:47 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/12 11:12:47 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/12 11:12:47 OGXReady: False - Component ManagementState is set to Removed 2026/07/12 11:12:47 RayReady: False - Component ManagementState is set to Removed 2026/07/12 11:12:47 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/12 11:12:47 TrainerReady: True - 2026/07/12 11:12:47 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/12 11:12:47 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/12 11:12:47 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/12 11:12:47 === RECENT EVENTS === 2026/07/12 11:12:47 11:12:27 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-b6mg7_119acb27-c8ef-4bc8-8287-b3be008cde8d became leader 2026/07/12 11:12:47 11:12:27 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-n58th_8f3654d4-d217-4015-9817-6392250e8d57 became leader 2026/07/12 11:12:47 11:12:09 INFO Pod/kserve-controller-manager-6c67dcb69c-b6mg7: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 880ms (880ms including waiting). Image size: 238435613 bytes. 2026/07/12 11:12:47 11:12:09 INFO Pod/llmisvc-controller-manager-5d65df97bb-n58th: Created - Created container: manager 2026/07/12 11:12:47 11:12:09 INFO Pod/kserve-controller-manager-6c67dcb69c-b6mg7: Started - Started container manager 2026/07/12 11:12:47 11:12:09 INFO Pod/llmisvc-controller-manager-5d65df97bb-n58th: Started - Started container manager 2026/07/12 11:12:47 11:12:09 INFO Pod/llmisvc-controller-manager-5d65df97bb-n58th: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 521ms (521ms including waiting). Image size: 236366599 bytes. 2026/07/12 11:12:47 11:12:09 INFO Pod/kserve-controller-manager-6c67dcb69c-b6mg7: Created - Created container: manager 2026/07/12 11:12:47 11:12:08 INFO Pod/llmisvc-controller-manager-5d65df97bb-n58th: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/12 11:12:47 11:12:08 INFO Pod/llmisvc-controller-manager-5d65df97bb-n58th: AddedInterface - Add eth0 [10.128.1.158/23] from ovn-kubernetes 2026/07/12 11:12:47 11:12:08 INFO Pod/kserve-controller-manager-6c67dcb69c-b6mg7: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/12 11:12:47 11:12:08 INFO Pod/kserve-controller-manager-6c67dcb69c-b6mg7: AddedInterface - Add eth0 [10.128.1.157/23] from ovn-kubernetes 2026/07/12 11:12:47 11:11:53 WARN Pod/llmisvc-controller-manager-5d65df97bb-n58th: FailedCreatePodSandBox - Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_llmisvc-controller-manager-5d65df97bb-n58th_redhat-ods-applications_bc6c420f-de77-441c-828f-3c5ba86c5f4d_0(57b59421359e8b9fc37300e8445d83081dcca343d556da6a0e8a16b32c69fe93): error adding pod redhat-ods-applications_llmisvc-controller-manager-5d65df97bb-n58th to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"57b59421359e8b9fc37300e8445d83081dcca343d556da6a0e8a16b32c69fe93" Netns:"/var/run/netns/14b7596c-fbaf-412f-b3ef-19fdd5f27172" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=redhat-ods-applications;K8S_POD_NAME=llmisvc-controller-manager-5d65df97bb-n58th;K8S_POD_INFRA_CONTAINER_ID=57b59421359e8b9fc37300e8445d83081dcca343d556da6a0e8a16b32c69fe93;K8S_POD_UID=bc6c420f-de77-441c-828f-3c5ba86c5f4d" Path:"" ERRORED: error configuring pod [redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-n58th] networking: Multus: [redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-n58th/bc6c420f-de77-441c-828f-3c5ba86c5f4d]: error waiting for pod: Get "https://api.rh-nightly-pm.hypershift.local:443/api/v1/namespaces/redhat-ods-applications/pods/llmisvc-controller-manager-5d65df97bb-n58th?timeout=1m0s": context deadline exceeded ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} 2026/07/12 11:12:47 11:11:52 WARN Pod/kserve-controller-manager-6c67dcb69c-b6mg7: FailedCreatePodSandBox - Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kserve-controller-manager-6c67dcb69c-b6mg7_redhat-ods-applications_f851c0f5-b081-4ade-829d-705494c29518_0(ca1f9e512fd30d28ce5fe4889ead7c586505dd91a4943500a1b1e73c73671288): error adding pod redhat-ods-applications_kserve-controller-manager-6c67dcb69c-b6mg7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"ca1f9e512fd30d28ce5fe4889ead7c586505dd91a4943500a1b1e73c73671288" Netns:"/var/run/netns/dcedcedb-7a2a-415b-a6be-64c7f0753bf6" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=redhat-ods-applications;K8S_POD_NAME=kserve-controller-manager-6c67dcb69c-b6mg7;K8S_POD_INFRA_CONTAINER_ID=ca1f9e512fd30d28ce5fe4889ead7c586505dd91a4943500a1b1e73c73671288;K8S_POD_UID=f851c0f5-b081-4ade-829d-705494c29518" Path:"" ERRORED: error configuring pod [redhat-ods-applications/kserve-controller-manager-6c67dcb69c-b6mg7] networking: Multus: [redhat-ods-applications/kserve-controller-manager-6c67dcb69c-b6mg7/f851c0f5-b081-4ade-829d-705494c29518]: error waiting for pod: Get "https://api.rh-nightly-pm.hypershift.local:443/api/v1/namespaces/redhat-ods-applications/pods/kserve-controller-manager-6c67dcb69c-b6mg7?timeout=1m0s": context deadline exceeded ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} 2026/07/12 11:12:47 11:11:43 WARN Pod/maas-controller-85bdcf9988-cpqkg: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-9hzjh); attempting to pull the image may not succeed. 2026/07/12 11:12:47 11:11:37 WARN Pod/kserve-controller-manager-6c67dcb69c-b6mg7: FailedCreatePodSandBox - Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kserve-controller-manager-6c67dcb69c-b6mg7_redhat-ods-applications_f851c0f5-b081-4ade-829d-705494c29518_0(a30357dcf215aa2cc3d57c76a14f991000a4533403ab7a8bff3214f785a403ac): error adding pod redhat-ods-applications_kserve-controller-manager-6c67dcb69c-b6mg7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"a30357dcf215aa2cc3d57c76a14f991000a4533403ab7a8bff3214f785a403ac" Netns:"/var/run/netns/5e55999d-bfe0-4d5f-94d9-2096f29cebd1" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=redhat-ods-applications;K8S_POD_NAME=kserve-controller-manager-6c67dcb69c-b6mg7;K8S_POD_INFRA_CONTAINER_ID=a30357dcf215aa2cc3d57c76a14f991000a4533403ab7a8bff3214f785a403ac;K8S_POD_UID=f851c0f5-b081-4ade-829d-705494c29518" Path:"" ERRORED: error configuring pod [redhat-ods-applications/kserve-controller-manager-6c67dcb69c-b6mg7] networking: Multus: [redhat-ods-applications/kserve-controller-manager-6c67dcb69c-b6mg7/f851c0f5-b081-4ade-829d-705494c29518]: error waiting for pod: Get "https://api.rh-nightly-pm.hypershift.local:443/api/v1/namespaces/redhat-ods-applications/pods/kserve-controller-manager-6c67dcb69c-b6mg7?timeout=1m0s": context deadline exceeded ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} 2026/07/12 11:12:47 11:11:37 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-nn272_03f2f5d5-6d0b-46f7-a86c-379acc2db80d became leader 2026/07/12 11:12:47 11:11:37 WARN Pod/llmisvc-controller-manager-5d65df97bb-n58th: FailedCreatePodSandBox - Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_llmisvc-controller-manager-5d65df97bb-n58th_redhat-ods-applications_bc6c420f-de77-441c-828f-3c5ba86c5f4d_0(f8760db1a00c1ec0f069f5f7f80b4dfd7f9e7d0514805a62ac07d5771d820135): error adding pod redhat-ods-applications_llmisvc-controller-manager-5d65df97bb-n58th to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"f8760db1a00c1ec0f069f5f7f80b4dfd7f9e7d0514805a62ac07d5771d820135" Netns:"/var/run/netns/fb681e78-ab14-4549-a48c-bc36806f1afc" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=redhat-ods-applications;K8S_POD_NAME=llmisvc-controller-manager-5d65df97bb-n58th;K8S_POD_INFRA_CONTAINER_ID=f8760db1a00c1ec0f069f5f7f80b4dfd7f9e7d0514805a62ac07d5771d820135;K8S_POD_UID=bc6c420f-de77-441c-828f-3c5ba86c5f4d" Path:"" ERRORED: error configuring pod [redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-n58th] networking: Multus: [redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-n58th/bc6c420f-de77-441c-828f-3c5ba86c5f4d]: error waiting for pod: Get "https://api.rh-nightly-pm.hypershift.local:443/api/v1/namespaces/redhat-ods-applications/pods/llmisvc-controller-manager-5d65df97bb-n58th?timeout=1m0s": context deadline exceeded ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} 2026/07/12 11:12:47 11:11:34 WARN Pod/kserve-controller-manager-6c67dcb69c-b6mg7: FailedCreatePodSandBox - Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kserve-controller-manager-6c67dcb69c-b6mg7_redhat-ods-applications_f851c0f5-b081-4ade-829d-705494c29518_0(2e4e4721f51d19053097b548ce2cb3f2097c5fe9210a62ac5383510c85dfe41c): error adding pod redhat-ods-applications_kserve-controller-manager-6c67dcb69c-b6mg7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"2e4e4721f51d19053097b548ce2cb3f2097c5fe9210a62ac5383510c85dfe41c" Netns:"/var/run/netns/317be462-c77c-4278-a9b5-9fd45568653f" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=redhat-ods-applications;K8S_POD_NAME=kserve-controller-manager-6c67dcb69c-b6mg7;K8S_POD_INFRA_CONTAINER_ID=2e4e4721f51d19053097b548ce2cb3f2097c5fe9210a62ac5383510c85dfe41c;K8S_POD_UID=f851c0f5-b081-4ade-829d-705494c29518" Path:"" ERRORED: error configuring pod [redhat-ods-applications/kserve-controller-manager-6c67dcb69c-b6mg7] networking: Multus: [redhat-ods-applications/kserve-controller-manager-6c67dcb69c-b6mg7/f851c0f5-b081-4ade-829d-705494c29518]: error waiting for pod: Get "https://api.rh-nightly-pm.hypershift.local:443/api/v1/namespaces/redhat-ods-applications/pods/kserve-controller-manager-6c67dcb69c-b6mg7?timeout=1m0s": context deadline exceeded ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} 2026/07/12 11:12:47 11:11:34 WARN Pod/llmisvc-controller-manager-5d65df97bb-n58th: FailedCreatePodSandBox - Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_llmisvc-controller-manager-5d65df97bb-n58th_redhat-ods-applications_bc6c420f-de77-441c-828f-3c5ba86c5f4d_0(78f7baa70f4aa5628837c1dad24c67ea3e74c54cd1bb25a4e564e8ebcc1b402c): error adding pod redhat-ods-applications_llmisvc-controller-manager-5d65df97bb-n58th to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"78f7baa70f4aa5628837c1dad24c67ea3e74c54cd1bb25a4e564e8ebcc1b402c" Netns:"/var/run/netns/4b0967c9-4142-45bb-8d21-1b6acd8784ae" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=redhat-ods-applications;K8S_POD_NAME=llmisvc-controller-manager-5d65df97bb-n58th;K8S_POD_INFRA_CONTAINER_ID=78f7baa70f4aa5628837c1dad24c67ea3e74c54cd1bb25a4e564e8ebcc1b402c;K8S_POD_UID=bc6c420f-de77-441c-828f-3c5ba86c5f4d" Path:"" ERRORED: error configuring pod [redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-n58th] networking: Multus: [redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-n58th/bc6c420f-de77-441c-828f-3c5ba86c5f4d]: error waiting for pod: Get "https://api.rh-nightly-pm.hypershift.local:443/api/v1/namespaces/redhat-ods-applications/pods/llmisvc-controller-manager-5d65df97bb-n58th?timeout=1m0s": context deadline exceeded ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} 2026/07/12 11:12:47 11:11:31 INFO Pod/llmisvc-controller-manager-5d65df97bb-n58th: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-n58th to ip-10-0-1-120.us-east-2.compute.internal 2026/07/12 11:12:47 11:11:31 INFO Pod/kserve-controller-manager-6c67dcb69c-b6mg7: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-b6mg7 to ip-10-0-1-120.us-east-2.compute.internal 2026/07/12 11:12:47 11:11:31 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/12 11:12:47 11:11:31 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/12 11:12:47 11:11:31 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-n58th 2026/07/12 11:12:47 11:11:31 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-b6mg7 2026/07/12 11:12:47 11:11:20 INFO Pod/model-serving-api-6688cd56bd-nb46b: Started - Started container server 2026/07/12 11:12:47 11:11:20 INFO Pod/model-serving-api-6688cd56bd-nb46b: Created - Created container: server 2026/07/12 11:12:47 11:11:20 INFO Pod/odh-model-controller-6849fc49d8-nn272: Started - Started container manager 2026/07/12 11:12:47 11:11:20 INFO Pod/odh-model-controller-6849fc49d8-nn272: Created - Created container: manager 2026/07/12 11:12:47 11:11:20 INFO Pod/odh-model-controller-6849fc49d8-nn272: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 474ms (474ms including waiting). Image size: 238959091 bytes. 2026/07/12 11:12:47 11:11:20 INFO Pod/model-serving-api-6688cd56bd-nb46b: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/12 11:12:47 11:11:19 INFO Pod/model-serving-api-6688cd56bd-nb46b: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-nb46b to ip-10-0-1-120.us-east-2.compute.internal 2026/07/12 11:12:47 11:11:19 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/12 11:12:47 11:11:19 INFO Pod/odh-model-controller-6849fc49d8-nn272: AddedInterface - Add eth0 [10.128.1.155/23] from ovn-kubernetes 2026/07/12 11:12:47 11:11:19 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/12 11:12:47 11:11:19 INFO Pod/model-serving-api-6688cd56bd-nb46b: AddedInterface - Add eth0 [10.128.1.156/23] from ovn-kubernetes 2026/07/12 11:12:47 11:11:19 INFO Pod/odh-model-controller-6849fc49d8-nn272: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/12 11:12:47 11:11:19 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/12 11:12:47 11:11:19 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-nb46b 2026/07/12 11:12:47 11:11:18 WARN Pod/odh-model-controller-6849fc49d8-nn272: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/12 11:12:47 11:11:17 INFO Pod/odh-model-controller-6849fc49d8-nn272: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-nn272 to ip-10-0-1-120.us-east-2.compute.internal 2026/07/12 11:12:47 11:11:17 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/12 11:12:47 11:11:17 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-nn272 2026/07/12 11:12:47 11:11:09 INFO Pod/model-serving-api-6688cd56bd-zrlp5: Killing - Stopping container server 2026/07/12 11:12:47 11:11:09 INFO Pod/odh-model-controller-6849fc49d8-j2nvg: Killing - Stopping container manager 2026/07/12 11:12:47 11:11:08 INFO Pod/llmisvc-controller-manager-5d65df97bb-dphbx: Killing - Stopping container manager 2026/07/12 11:12:47 11:11:08 INFO Pod/kserve-controller-manager-6c67dcb69c-7r2nv: Killing - Stopping container manager 2026/07/12 11:12:47 11:10:51 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-dphbx_32e5f95a-3b2a-4734-b60f-f78e84d07886 became leader 2026/07/12 11:12:47 11:10:47 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-7r2nv_8209c306-76d0-44bc-bc75-6f9dd95330f5 became leader 2026/07/12 11:12:47 11:10:36 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-j2nvg_75994d4f-cdf3-4e1e-9056-860580f2eca5 became leader 2026/07/12 11:12:47 11:10:31 INFO Pod/llmisvc-controller-manager-5d65df97bb-dphbx: Created - Created container: manager 2026/07/12 11:12:47 11:10:31 INFO Pod/llmisvc-controller-manager-5d65df97bb-dphbx: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/12 11:12:47 11:10:31 INFO Pod/llmisvc-controller-manager-5d65df97bb-dphbx: Started - Started container manager 2026/07/12 11:12:47 11:10:31 INFO Pod/llmisvc-controller-manager-5d65df97bb-dphbx: AddedInterface - Add eth0 [10.128.1.153/23] from ovn-kubernetes 2026/07/12 11:12:47 11:10:31 INFO Pod/kserve-controller-manager-6c67dcb69c-7r2nv: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 502ms (502ms including waiting). Image size: 238435613 bytes. 2026/07/12 11:12:47 11:10:31 INFO Pod/llmisvc-controller-manager-5d65df97bb-dphbx: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 592ms (592ms including waiting). Image size: 236366599 bytes. 2026/07/12 11:12:47 11:10:31 INFO Pod/kserve-controller-manager-6c67dcb69c-7r2nv: Created - Created container: manager 2026/07/12 11:12:47 11:10:31 INFO Pod/kserve-controller-manager-6c67dcb69c-7r2nv: Started - Started container manager 2026/07/12 11:12:47 11:10:30 INFO Pod/llmisvc-controller-manager-5d65df97bb-dphbx: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-dphbx to ip-10-0-1-120.us-east-2.compute.internal 2026/07/12 11:12:47 11:10:30 INFO Pod/kserve-controller-manager-6c67dcb69c-7r2nv: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-7r2nv to ip-10-0-1-120.us-east-2.compute.internal 2026/07/12 11:12:47 11:10:30 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-7r2nv 2026/07/12 11:12:47 11:10:30 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-7ccfdd66-lbnjq_97c8aee6-f7c0-4cb3-93b7-3209275950ed became leader 2026/07/12 11:12:47 11:10:30 WARN Pod/llmisvc-controller-manager-5d65df97bb-dphbx: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 2026/07/12 11:12:47 11:10:30 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/12 11:12:47 11:10:30 INFO Pod/kserve-controller-manager-6c67dcb69c-7r2nv: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/12 11:12:47 11:10:30 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/12 11:12:47 11:10:30 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-dphbx 2026/07/12 11:12:47 11:10:30 INFO Pod/kserve-controller-manager-6c67dcb69c-7r2nv: AddedInterface - Add eth0 [10.128.1.152/23] from ovn-kubernetes 2026/07/12 11:12:47 11:10:20 INFO Pod/model-serving-api-6688cd56bd-zrlp5: AddedInterface - Add eth0 [10.128.1.151/23] from ovn-kubernetes 2026/07/12 11:12:47 11:10:20 INFO Pod/model-serving-api-6688cd56bd-zrlp5: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/12 11:12:47 11:10:20 INFO Pod/model-serving-api-6688cd56bd-zrlp5: Created - Created container: server 2026/07/12 11:12:47 11:10:20 INFO Pod/model-serving-api-6688cd56bd-zrlp5: Started - Started container server 2026/07/12 11:12:47 11:10:19 INFO Pod/model-serving-api-6688cd56bd-zrlp5: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-zrlp5 to ip-10-0-1-120.us-east-2.compute.internal 2026/07/12 11:12:47 11:10:19 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/12 11:12:47 11:10:19 INFO Pod/odh-model-controller-6849fc49d8-j2nvg: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 523ms (523ms including waiting). Image size: 238959091 bytes. 2026/07/12 11:12:47 11:10:19 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/12 11:12:47 11:10:19 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-zrlp5 2026/07/12 11:12:47 11:10:19 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/12 11:12:47 11:10:19 INFO Pod/odh-model-controller-6849fc49d8-j2nvg: Started - Started container manager 2026/07/12 11:12:47 11:10:19 INFO Pod/odh-model-controller-6849fc49d8-j2nvg: Created - Created container: manager 2026/07/12 11:12:47 11:10:18 INFO Pod/odh-model-controller-6849fc49d8-j2nvg: AddedInterface - Add eth0 [10.128.1.150/23] from ovn-kubernetes 2026/07/12 11:12:47 11:10:18 INFO Pod/odh-model-controller-6849fc49d8-j2nvg: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/12 11:12:47 11:10:17 INFO Pod/odh-model-controller-6849fc49d8-j2nvg: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-j2nvg to ip-10-0-1-120.us-east-2.compute.internal 2026/07/12 11:12:47 11:10:17 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-j2nvg 2026/07/12 11:12:47 11:10:17 WARN Pod/odh-model-controller-6849fc49d8-j2nvg: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/12 11:12:47 11:10:17 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/12 11:12:47 11:10:12 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-lbnjq: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-7ccfdd66-lbnjq to ip-10-0-1-120.us-east-2.compute.internal 2026/07/12 11:12:47 11:10:12 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-lbnjq: Pulled - Container image "quay.io/rhoai/odh-trainer-rhel9@sha256:59ea523e2df9c1e705c755308ac11420682081cc63d54890e2b15fbd7d8ba570" already present on machine 2026/07/12 11:12:47 11:10:12 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-lbnjq: AddedInterface - Add eth0 [10.128.1.149/23] from ovn-kubernetes 2026/07/12 11:12:47 11:10:12 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-lbnjq: Created - Created container: manager 2026/07/12 11:12:47 11:10:12 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-7ccfdd66 from 0 to 1 2026/07/12 11:12:47 11:10:12 INFO ReplicaSet/kubeflow-trainer-controller-manager-7ccfdd66: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-7ccfdd66-lbnjq 2026/07/12 11:12:47 11:10:12 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-lbnjq: Started - Started container manager 2026/07/12 11:12:47 11:10:11 WARN ReplicaSet/llmisvc-controller-manager-5d65df97bb: FailedCreate - Error creating: pods "llmisvc-controller-manager-5d65df97bb-" is forbidden: error looking up service account redhat-ods-applications/llmisvc-controller-manager: serviceaccount "llmisvc-controller-manager" not found 2026/07/12 11:12:47 11:10:11 WARN ReplicaSet/kserve-controller-manager-6c67dcb69c: FailedCreate - Error creating: pods "kserve-controller-manager-6c67dcb69c-" is forbidden: error looking up service account redhat-ods-applications/kserve-controller-manager: serviceaccount "kserve-controller-manager" not found 2026/07/12 11:12:47 11:10:10 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/12 11:12:47 11:10:10 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/12 11:12:47 11:10:07 INFO Pod/llmisvc-controller-manager-5d65df97bb-7zsl6: Killing - Stopping container manager 2026/07/12 11:12:47 11:10:07 INFO Pod/odh-model-controller-6849fc49d8-mbk8w: Killing - Stopping container manager 2026/07/12 11:12:47 11:10:07 INFO Pod/model-serving-api-6688cd56bd-nz5ff: Killing - Stopping container server 2026/07/12 11:12:47 11:10:07 INFO Pod/kserve-controller-manager-6c67dcb69c-d279d: Killing - Stopping container manager 2026/07/12 11:12:47 11:08:40 INFO Pod/maas-controller-85bdcf9988-cpqkg: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/12 11:12:47 11:08:40 INFO Pod/maas-controller-85bdcf9988-cpqkg: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 525ms (525ms including waiting). Image size: 212866077 bytes. 2026/07/12 11:12:47 === RESOURCE QUOTAS === 2026/07/12 11:12:47 No resource quotas found in monitored namespaces 2026/07/12 11:12:47 Classification for components: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/12 11:12:47 Evidence: container maas-controller-85bdcf9988-cpqkg/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-cpqkg_redhat-ods-applications(81519e5d-224d-4651-b3b4-dd63ce584001) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-85bdcf9988-cpqkg/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-cpqkg_redhat-ods-applications(81519e5d-224d-4651-b3b4-dd63ce584001)"],"confidence":"medium"} 2026/07/12 11:12:47 Diagnostics complete for failed test: components controller_test.go:484: Stopping: components test failed. 2026/07/12 11:12:47 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/12 11:12:47 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/12 11:12:49 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (1527.21s)