=== RUN TestOdhOperator/components/group_1/kserve/Validate_component_spec testing.go:1811: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_1/kserve/Validate_component_spec (600.03s) === RUN TestOdhOperator/components/group_1/kserve === PAUSE TestOdhOperator/components/group_1/kserve === CONT TestOdhOperator/components/group_1/kserve test_context_test.go:223: Timed out after 600.008s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:114 with: Expected resource 'default-kserve' of kind 'Kserve' to match condition {.spec.nim.managementState == "" []}. Expected <string>: map[apiVersion:components.platform.opendatahub.io/v1alpha1 kind:Kserve metadata:map[annotations:map[component.opendatahub.io/management-state:Managed platform.opendatahub.io/instance.generation:169 platform.opendatahub.io/instance.name:default-dsc platform.opendatahub.io/instance.uid:9013f03c-d6c8-43c1-bdd4-c3e666f6094f platform.opendatahub.io/type:OpenShift AI Self-Managed platform.opendatahub.io/version:3.5.0-ea.1] creationTimestamp:2026-07-15T08:09:15Z generation:1 labels:map[platform.opendatahub.io/part-of:datasciencecluster] managedFields:[map[apiVersion:components.platform.opendatahub.io/v1alpha1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:component.opendatahub.io/management-state:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:labels:map[f:platform.opendatahub.io/part-of:map[]] f:ownerReferences:map[k:{"uid":"9013f03c-d6c8-43c1-bdd4-c3e666f6094f"}:map[]]] f:spec:map[f:modelsAsService:map[f:managementState:map[]] f:nim:map[] f:rawDeploymentServiceConfig:map[] f:wva:map[f:managementState:map[]]]] manager:datasciencecluster operation:Apply time:2026-07-15T08:11:14Z] map[apiVersion:components.platform.opendatahub.io/v1alpha1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:component.opendatahub.io/management-state:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:labels:map[f:platform.opendatahub.io/part-of:map[]] f:ownerReferences:map[k:{"uid":"9013f03c-d6c8-43c1-bdd4-c3e666f6094f"}:map[]]] f:status:map[f:conditions:map[] f:observedGeneration:map[] f:phase:map[] f:releases:map[k:{"name":"KServe"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-inference-scheduler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-workload-variant-autoscaler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"vLLM"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]]] manager:kserve operation:Apply subresource:status time:2026-07-15T08:20:01Z]] name:default-kserve ownerReferences:[map[apiVersion:datasciencecluster.opendatahub.io/v2 blockOwnerDeletion:true controller:true kind:DataScienceCluster name:default-dsc uid:9013f03c-d6c8-43c1-bdd4-c3e666f6094f]] resourceVersion:33392298 uid:faee2ca1-9efa-4680-ad03-bb29e759e532] spec:map[modelsAsService:map[managementState:Managed] nim:map[airGapped:false managementState:Managed] rawDeploymentServiceConfig:Headless wva:map[managementState:Removed]] status:map[conditions:[map[lastTransitionTime:2026-07-15T08:20:01Z status:True type:Ready] map[lastTransitionTime:2026-07-15T08:20:01Z observedGeneration:1 status:True type:ProvisioningSucceeded] map[lastTransitionTime:2026-07-15T08:20:01Z observedGeneration:1 status:True type:DeploymentsAvailable] map[lastTransitionTime:2026-07-15T08:20:01Z status:True type:DependenciesAvailable] map[lastTransitionTime:2026-07-15T08:20:01Z status:True type:KserveLLMInferenceServiceDependencies] map[lastTransitionTime:2026-07-15T08:20:01Z status:True type:KserveLLMInferenceServiceWideEPDependencies]] observedGeneration:1 phase:Ready releases:[map[name:KServe repoUrl:https://github.com/kserve/kserve/ version:v0.17.0] map[name:vLLM repoUrl:https://github.com/vllm-project/vllm version:v0.18.0] map[name:llm-d-inference-scheduler repoUrl:https://github.com/llm-d/llm-d-inference-scheduler version:v0.7.1] map[name:llm-d-workload-variant-autoscaler repoUrl:https://github.com/llm-d/llm-d-workload-variant-autoscaler version:v0.6.0]]]] to match expression <string>: .spec.nim.managementState == "" --- FAIL: TestOdhOperator/components/group_1/kserve (670.91s) === RUN TestOdhOperator/components/group_1 --- FAIL: TestOdhOperator/components/group_1 (0.00s) === RUN TestOdhOperator/components 2026/07/15 08:20:14 TEST FAILURE DETECTED: group 1 2026/07/15 08:20:14 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 1) === 2026/07/15 08:20:15 === CLUSTER STATE === 2026/07/15 08:20:15 Node: ip-10-0-1-23.us-east-2.compute.internal 2026/07/15 08:20:15 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 08:20:15 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 08:20:15 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 08:20:15 Condition Ready: True - kubelet is posting ready status 2026/07/15 08:20:15 Resources: 15500m CPU, 63182900Ki memory allocatable (of 16000m CPU, 64333876Ki memory capacity) 2026/07/15 08:20:15 Node: ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 08:20:15 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 08:20:15 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 08:20:15 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 08:20:15 Condition Ready: True - kubelet is posting ready status 2026/07/15 08:20:15 Resources: 15500m CPU, 63871012Ki memory allocatable (of 16000m CPU, 65021988Ki memory capacity) 2026/07/15 08:20:15 Node: ip-10-0-1-62.us-east-2.compute.internal 2026/07/15 08:20:15 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 08:20:15 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 08:20:15 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 08:20:15 Condition Ready: True - kubelet is posting ready status 2026/07/15 08:20:15 Resources: 15500m CPU, 63182896Ki memory allocatable (of 16000m CPU, 64333872Ki memory capacity) 2026/07/15 08:20:15 === DEPLOYMENTS === 2026/07/15 08:20:15 Namespace: redhat-ods-applications 2026/07/15 08:20:15 Deployment dashboard-redirect: 2/2 (READY) 2026/07/15 08:20:15 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/15 08:20:15 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/15 08:20:15 Deployment maas-api: 1/1 (READY) 2026/07/15 08:20:15 Deployment maas-controller: 1/1 (READY) 2026/07/15 08:20:15 Deployment maas-postgres: 1/1 (READY) 2026/07/15 08:20:15 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/15 08:20:15 Deployment model-serving-api: 1/1 (READY) 2026/07/15 08:20:15 Deployment odh-model-controller: 1/1 (READY) 2026/07/15 08:20:15 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/15 08:20:15 Namespace: kube-system 2026/07/15 08:20:15 === PODS === 2026/07/15 08:20:15 Failed to collect pod data: redhat-ods-applications/trustyai-service-operator-controller-manager-bc99cf85b-j5nns: container manager restarts=2 2026/07/15 08:20:15 === OPERATOR STATUS === 2026/07/15 08:20:15 Operator deployment: 3/3 ready 2026/07/15 08:20:15 === DSCI STATUS === 2026/07/15 08:20:15 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/15 08:20:15 === DSC STATUS === 2026/07/15 08:20:15 DSC default-dsc: 2026/07/15 08:20:15 Ready: True - 2026/07/15 08:20:15 ProvisioningSucceeded: True - 2026/07/15 08:20:15 ComponentsReady: True - 2026/07/15 08:20:15 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/15 08:20:15 DashboardReady: False - Component ManagementState is set to Removed 2026/07/15 08:20:15 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 08:20:15 KserveLLMInferenceServiceDependencies: True - 2026/07/15 08:20:15 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/15 08:20:15 KserveReady: True - 2026/07/15 08:20:15 KueueReady: True - 2026/07/15 08:20:15 MLflowOperatorReady: True - 2026/07/15 08:20:15 ModelControllerReady: True - 2026/07/15 08:20:15 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/15 08:20:15 ModelsAsServiceReady: True - MaaS platform manifests applied and maas-api deployment is available 2026/07/15 08:20:15 OGXReady: False - Component ManagementState is set to Removed 2026/07/15 08:20:15 RayReady: False - Component ManagementState is set to Removed 2026/07/15 08:20:15 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 08:20:15 TrainerReady: False - Component ManagementState is set to Removed 2026/07/15 08:20:15 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 08:20:15 TrustyAIReady: True - 2026/07/15 08:20:15 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/15 08:20:15 === RECENT EVENTS === 2026/07/15 08:20:15 No recent events found in monitored namespaces 2026/07/15 08:20:15 === RESOURCE QUOTAS === 2026/07/15 08:20:15 No resource quotas found in monitored namespaces 2026/07/15 08:20:15 Classification for group 1: infrastructure/dsci-unhealthy (code=1009, confidence=medium) 2026/07/15 08:20:15 Evidence: DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"dsci-unhealthy","error_code":1009,"evidence":["DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False"],"confidence":"medium"} 2026/07/15 08:20:15 Diagnostics complete for failed test: group 1 controller_test.go:752: Stopping: group 1 test failed. 2026/07/15 08:20:15 Circuit breaker: failure classified as infrastructure/dsci-unhealthy (code=1009) 2026/07/15 08:20:15 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/15 08:23:48 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (1437.68s) === RUN TestOdhOperator 2026/07/15 08:07:11 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/15 08:31:19 TEST FAILURE DETECTED: components 2026/07/15 08:31:19 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/15 08:31:20 === CLUSTER STATE === 2026/07/15 08:31:20 Node: ip-10-0-1-23.us-east-2.compute.internal 2026/07/15 08:31:20 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 08:31:20 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 08:31:20 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 08:31:20 Condition Ready: True - kubelet is posting ready status 2026/07/15 08:31:20 Resources: 15500m CPU, 63182900Ki memory allocatable (of 16000m CPU, 64333876Ki memory capacity) 2026/07/15 08:31:20 Node: ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 08:31:20 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 08:31:20 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 08:31:20 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 08:31:20 Condition Ready: True - kubelet is posting ready status 2026/07/15 08:31:20 Resources: 15500m CPU, 63871012Ki memory allocatable (of 16000m CPU, 65021988Ki memory capacity) 2026/07/15 08:31:20 Node: ip-10-0-1-62.us-east-2.compute.internal 2026/07/15 08:31:20 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 08:31:20 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 08:31:20 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 08:31:20 Condition Ready: True - kubelet is posting ready status 2026/07/15 08:31:20 Resources: 15500m CPU, 63182896Ki memory allocatable (of 16000m CPU, 64333872Ki memory capacity) 2026/07/15 08:31:20 === DEPLOYMENTS === 2026/07/15 08:31:20 Namespace: redhat-ods-applications 2026/07/15 08:31:20 Deployment dashboard-redirect: 2/2 (READY) 2026/07/15 08:31:20 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/15 08:31:20 Deployment kubeflow-trainer-controller-manager: 1/1 (READY) 2026/07/15 08:31:20 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/15 08:31:20 Deployment maas-postgres: 1/1 (READY) 2026/07/15 08:31:20 Deployment model-serving-api: 1/1 (READY) 2026/07/15 08:31:20 Deployment odh-model-controller: 1/1 (READY) 2026/07/15 08:31:20 Namespace: kube-system 2026/07/15 08:31:20 === PODS === 2026/07/15 08:31:20 Namespace redhat-ods-applications: no problematic pods found 2026/07/15 08:31:20 Namespace kube-system: no problematic pods found 2026/07/15 08:31:20 === OPERATOR STATUS === 2026/07/15 08:31:20 Operator deployment: 3/3 ready 2026/07/15 08:31:20 === DSCI STATUS === 2026/07/15 08:31:20 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/15 08:31:20 === DSC STATUS === 2026/07/15 08:31:20 DSC default-dsc: 2026/07/15 08:31:20 Ready: True - 2026/07/15 08:31:20 ProvisioningSucceeded: True - 2026/07/15 08:31:20 ComponentsReady: True - 2026/07/15 08:31:20 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/15 08:31:20 DashboardReady: False - Component ManagementState is set to Removed 2026/07/15 08:31:20 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 08:31:20 KserveLLMInferenceServiceDependencies: True - 2026/07/15 08:31:20 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/15 08:31:20 KserveReady: True - 2026/07/15 08:31:20 KueueReady: False - Component ManagementState is set to Removed 2026/07/15 08:31:20 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 08:31:20 ModelControllerReady: True - 2026/07/15 08:31:20 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/15 08:31:20 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/15 08:31:20 OGXReady: False - Component ManagementState is set to Removed 2026/07/15 08:31:20 RayReady: False - Component ManagementState is set to Removed 2026/07/15 08:31:20 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 08:31:20 TrainerReady: True - 2026/07/15 08:31:20 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 08:31:20 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/15 08:31:20 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/15 08:31:20 === RECENT EVENTS === 2026/07/15 08:31:20 08:31:04 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-7747874bf-cx9vc_f2a8311a-5e85-48fe-840e-3d9fcf16baad became leader 2026/07/15 08:31:20 08:31:04 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-78dcbbdc6c-cpp9l_5e8552ba-447a-43f7-982b-b0ab89b9d35d became leader 2026/07/15 08:31:20 08:30:50 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-5b978c765f-dz4sq_b33aacd8-84bb-4eab-a0ff-c7dda7bb0cfd became leader 2026/07/15 08:31:20 08:30:46 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-cpp9l: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" 2026/07/15 08:31:20 08:30:46 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-cpp9l: Started - Started container manager 2026/07/15 08:31:20 08:30:46 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-cpp9l: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" in 623ms (623ms including waiting). Image size: 235181729 bytes. 2026/07/15 08:31:20 08:30:46 INFO Pod/kserve-controller-manager-7747874bf-cx9vc: Created - Created container: manager 2026/07/15 08:31:20 08:30:46 INFO Pod/kserve-controller-manager-7747874bf-cx9vc: Started - Started container manager 2026/07/15 08:31:20 08:30:46 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-cpp9l: Created - Created container: manager 2026/07/15 08:31:20 08:30:45 INFO Pod/kserve-controller-manager-7747874bf-cx9vc: AddedInterface - Add eth0 [10.130.1.97/23] from ovn-kubernetes 2026/07/15 08:31:20 08:30:45 INFO Pod/kserve-controller-manager-7747874bf-cx9vc: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" 2026/07/15 08:31:20 08:30:45 INFO Pod/kserve-controller-manager-7747874bf-cx9vc: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" in 576ms (576ms including waiting). Image size: 237680312 bytes. 2026/07/15 08:31:20 08:30:45 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-cpp9l: AddedInterface - Add eth0 [10.129.1.100/23] from ovn-kubernetes 2026/07/15 08:31:20 08:30:44 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-cpp9l: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-78dcbbdc6c-cpp9l to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 08:31:20 08:30:44 WARN Pod/llmisvc-controller-manager-78dcbbdc6c-cpp9l: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 2026/07/15 08:31:20 08:30:44 WARN Pod/kserve-controller-manager-7747874bf-cx9vc: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 2026/07/15 08:31:20 08:30:43 INFO Pod/kserve-controller-manager-7747874bf-cx9vc: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-7747874bf-cx9vc to ip-10-0-1-23.us-east-2.compute.internal 2026/07/15 08:31:20 08:30:43 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-78dcbbdc6c from 0 to 1 2026/07/15 08:31:20 08:30:43 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7747874bf from 0 to 1 2026/07/15 08:31:20 08:30:43 INFO ReplicaSet/kserve-controller-manager-7747874bf: SuccessfulCreate - Created pod: kserve-controller-manager-7747874bf-cx9vc 2026/07/15 08:31:20 08:30:43 INFO ReplicaSet/llmisvc-controller-manager-78dcbbdc6c: SuccessfulCreate - Created pod: llmisvc-controller-manager-78dcbbdc6c-cpp9l 2026/07/15 08:31:20 08:30:32 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 08:31:20 08:30:32 INFO Pod/model-serving-api-67c44bccdf-6rdzq: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-67c44bccdf-6rdzq to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 08:31:20 08:30:32 INFO Pod/odh-model-controller-5b978c765f-dz4sq: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" in 527ms (527ms including waiting). Image size: 238719886 bytes. 2026/07/15 08:31:20 08:30:32 INFO Pod/model-serving-api-67c44bccdf-6rdzq: AddedInterface - Add eth0 [10.129.1.99/23] from ovn-kubernetes 2026/07/15 08:31:20 08:30:32 INFO Pod/model-serving-api-67c44bccdf-6rdzq: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:226370d82da697958a0cd64b5cfa82f290ae1bb701e74b72739fcac0b38a834d" already present on machine 2026/07/15 08:31:20 08:30:32 INFO Pod/model-serving-api-67c44bccdf-6rdzq: Created - Created container: server 2026/07/15 08:31:20 08:30:32 INFO Pod/model-serving-api-67c44bccdf-6rdzq: Started - Started container server 2026/07/15 08:31:20 08:30:32 INFO ReplicaSet/model-serving-api-67c44bccdf: SuccessfulCreate - Created pod: model-serving-api-67c44bccdf-6rdzq 2026/07/15 08:31:20 08:30:32 INFO Pod/odh-model-controller-5b978c765f-dz4sq: Created - Created container: manager 2026/07/15 08:31:20 08:30:32 INFO Pod/odh-model-controller-5b978c765f-dz4sq: Started - Started container manager 2026/07/15 08:31:20 08:30:31 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 08:31:20 08:30:31 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-67c44bccdf from 0 to 1 2026/07/15 08:31:20 08:30:31 INFO Pod/odh-model-controller-5b978c765f-dz4sq: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" 2026/07/15 08:31:20 08:30:31 INFO Pod/odh-model-controller-5b978c765f-dz4sq: AddedInterface - Add eth0 [10.129.1.98/23] from ovn-kubernetes 2026/07/15 08:31:20 08:30:31 WARN ReplicaSet/model-serving-api-67c44bccdf: FailedCreate - Error creating: pods "model-serving-api-67c44bccdf-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/15 08:31:20 08:30:30 INFO Pod/odh-model-controller-5b978c765f-dz4sq: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-5b978c765f-dz4sq to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 08:31:20 08:30:30 INFO ReplicaSet/odh-model-controller-5b978c765f: SuccessfulCreate - Created pod: odh-model-controller-5b978c765f-dz4sq 2026/07/15 08:31:20 08:30:30 WARN Pod/odh-model-controller-5b978c765f-dz4sq: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/15 08:31:20 08:30:30 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-5b978c765f from 0 to 1 2026/07/15 08:31:20 08:30:20 INFO Pod/odh-model-controller-5b978c765f-2nd2n: Killing - Stopping container manager 2026/07/15 08:31:20 08:30:20 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-ksscb: Killing - Stopping container manager 2026/07/15 08:31:20 08:30:20 INFO Pod/kserve-controller-manager-7747874bf-qsrfd: Killing - Stopping container manager 2026/07/15 08:31:20 08:30:20 INFO Pod/model-serving-api-67c44bccdf-vc9rd: Killing - Stopping container server 2026/07/15 08:31:20 08:30:19 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 08:31:20 08:30:03 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-78dcbbdc6c-ksscb_e51469cd-5a1a-4f6b-87e9-0510ab04776f became leader 2026/07/15 08:31:20 08:30:00 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-7747874bf-qsrfd_2912d84c-89c7-44e7-9aac-46e22ac700d2 became leader 2026/07/15 08:31:20 08:29:50 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-5b978c765f-2nd2n_f286dbc5-c11d-46ac-ab6e-75e14b086177 became leader 2026/07/15 08:31:20 08:29:44 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-ksscb: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" in 575ms (575ms including waiting). Image size: 235181729 bytes. 2026/07/15 08:31:20 08:29:44 INFO Pod/kserve-controller-manager-7747874bf-qsrfd: Started - Started container manager 2026/07/15 08:31:20 08:29:44 INFO Pod/kserve-controller-manager-7747874bf-qsrfd: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" in 609ms (609ms including waiting). Image size: 237680312 bytes. 2026/07/15 08:31:20 08:29:44 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-ksscb: Started - Started container manager 2026/07/15 08:31:20 08:29:44 INFO Pod/kserve-controller-manager-7747874bf-qsrfd: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" 2026/07/15 08:31:20 08:29:44 INFO Pod/kserve-controller-manager-7747874bf-qsrfd: AddedInterface - Add eth0 [10.129.1.97/23] from ovn-kubernetes 2026/07/15 08:31:20 08:29:44 INFO Pod/kserve-controller-manager-7747874bf-qsrfd: Created - Created container: manager 2026/07/15 08:31:20 08:29:44 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-ksscb: Created - Created container: manager 2026/07/15 08:31:20 08:29:43 WARN Pod/kserve-controller-manager-7747874bf-qsrfd: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 2026/07/15 08:31:20 08:29:43 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-ksscb: AddedInterface - Add eth0 [10.130.1.96/23] from ovn-kubernetes 2026/07/15 08:31:20 08:29:43 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-ksscb: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" 2026/07/15 08:31:20 08:29:42 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-ksscb: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-78dcbbdc6c-ksscb to ip-10-0-1-23.us-east-2.compute.internal 2026/07/15 08:31:20 08:29:42 INFO Pod/kserve-controller-manager-7747874bf-qsrfd: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-7747874bf-qsrfd to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 08:31:20 08:29:42 INFO ReplicaSet/kserve-controller-manager-7747874bf: SuccessfulCreate - Created pod: kserve-controller-manager-7747874bf-qsrfd 2026/07/15 08:31:20 08:29:42 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7747874bf from 0 to 1 2026/07/15 08:31:20 08:29:42 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-78dcbbdc6c from 0 to 1 2026/07/15 08:31:20 08:29:42 INFO ReplicaSet/llmisvc-controller-manager-78dcbbdc6c: SuccessfulCreate - Created pod: llmisvc-controller-manager-78dcbbdc6c-ksscb 2026/07/15 08:31:20 08:29:40 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-7dbc595499-59vqh_49d41832-e7dc-47e8-8129-8139db41d95f became leader 2026/07/15 08:31:20 08:29:32 INFO Pod/odh-model-controller-5b978c765f-2nd2n: Started - Started container manager 2026/07/15 08:31:20 08:29:32 INFO Pod/odh-model-controller-5b978c765f-2nd2n: Created - Created container: manager 2026/07/15 08:31:20 08:29:32 INFO Pod/odh-model-controller-5b978c765f-2nd2n: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" in 465ms (465ms including waiting). Image size: 238719886 bytes. 2026/07/15 08:31:20 08:29:31 INFO Pod/model-serving-api-67c44bccdf-vc9rd: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-67c44bccdf-vc9rd to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 08:31:20 08:29:31 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 08:31:20 08:29:31 INFO Pod/model-serving-api-67c44bccdf-vc9rd: AddedInterface - Add eth0 [10.129.1.96/23] from ovn-kubernetes 2026/07/15 08:31:20 08:29:31 INFO Pod/model-serving-api-67c44bccdf-vc9rd: Created - Created container: server 2026/07/15 08:31:20 08:29:31 INFO Pod/model-serving-api-67c44bccdf-vc9rd: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:226370d82da697958a0cd64b5cfa82f290ae1bb701e74b72739fcac0b38a834d" already present on machine 2026/07/15 08:31:20 08:29:31 INFO Pod/model-serving-api-67c44bccdf-vc9rd: Started - Started container server 2026/07/15 08:31:20 08:29:31 WARN ReplicaSet/model-serving-api-67c44bccdf: FailedCreate - Error creating: pods "model-serving-api-67c44bccdf-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/15 08:31:20 08:29:31 INFO ReplicaSet/model-serving-api-67c44bccdf: SuccessfulCreate - Created pod: model-serving-api-67c44bccdf-vc9rd 2026/07/15 08:31:20 08:29:31 INFO Pod/odh-model-controller-5b978c765f-2nd2n: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" 2026/07/15 08:31:20 08:29:31 INFO Pod/odh-model-controller-5b978c765f-2nd2n: AddedInterface - Add eth0 [10.129.1.95/23] from ovn-kubernetes 2026/07/15 08:31:20 08:29:30 WARN Pod/odh-model-controller-5b978c765f-2nd2n: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/15 08:31:20 08:29:30 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-67c44bccdf from 0 to 1 2026/07/15 08:31:20 08:29:29 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 08:31:20 08:29:29 INFO Pod/odh-model-controller-5b978c765f-2nd2n: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-5b978c765f-2nd2n to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 08:31:20 08:29:29 INFO ReplicaSet/odh-model-controller-5b978c765f: SuccessfulCreate - Created pod: odh-model-controller-5b978c765f-2nd2n 2026/07/15 08:31:20 08:29:29 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-5b978c765f from 0 to 1 2026/07/15 08:31:20 08:29:24 INFO Pod/kubeflow-trainer-controller-manager-7dbc595499-59vqh: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-7dbc595499-59vqh to ip-10-0-1-62.us-east-2.compute.internal 2026/07/15 08:31:20 08:29:24 INFO Pod/kubeflow-trainer-controller-manager-7dbc595499-59vqh: Created - Created container: manager 2026/07/15 08:31:20 08:29:24 WARN ReplicaSet/kserve-controller-manager-7747874bf: FailedCreate - Error creating: pods "kserve-controller-manager-7747874bf-" is forbidden: error looking up service account redhat-ods-applications/kserve-controller-manager: serviceaccount "kserve-controller-manager" not found 2026/07/15 08:31:20 08:29:24 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7747874bf from 0 to 1 2026/07/15 08:31:20 08:29:24 WARN ReplicaSet/llmisvc-controller-manager-78dcbbdc6c: FailedCreate - Error creating: pods "llmisvc-controller-manager-78dcbbdc6c-" is forbidden: error looking up service account redhat-ods-applications/llmisvc-controller-manager: serviceaccount "llmisvc-controller-manager" not found 2026/07/15 08:31:20 08:29:24 INFO Pod/kubeflow-trainer-controller-manager-7dbc595499-59vqh: Pulled - Container image "quay.io/rhoai/odh-trainer-rhel9@sha256:595f456a6b3900dd112fa04416a2d91f448f8d68031b0c2de0d1dcb5fed28aa1" already present on machine 2026/07/15 08:31:20 08:29:24 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-78dcbbdc6c from 0 to 1 2026/07/15 08:31:20 08:29:24 INFO Pod/kubeflow-trainer-controller-manager-7dbc595499-59vqh: AddedInterface - Add eth0 [10.128.1.245/23] from ovn-kubernetes 2026/07/15 08:31:20 08:29:24 INFO Pod/kubeflow-trainer-controller-manager-7dbc595499-59vqh: Started - Started container manager 2026/07/15 08:31:20 08:29:24 INFO ReplicaSet/kubeflow-trainer-controller-manager-7dbc595499: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-7dbc595499-59vqh 2026/07/15 08:31:20 08:29:24 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-7dbc595499 from 0 to 1 2026/07/15 08:31:20 08:29:20 INFO Pod/model-serving-api-67c44bccdf-g95ks: Killing - Stopping container server 2026/07/15 08:31:20 08:29:20 INFO Pod/kserve-controller-manager-7747874bf-ffgt6: Killing - Stopping container manager 2026/07/15 08:31:20 08:29:20 INFO Pod/odh-model-controller-5b978c765f-7zxff: Killing - Stopping container manager 2026/07/15 08:31:20 08:29:20 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-6ng4w: Killing - Stopping container manager 2026/07/15 08:31:20 08:29:19 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 08:31:20 08:29:09 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 08:31:20 08:29:09 INFO Pod/maas-api-5db58d77cb-dqzbp: Killing - Stopping container maas-api 2026/07/15 08:31:20 08:29:09 INFO Pod/maas-controller-85c784d7b6-dzmw2: Killing - Stopping container manager 2026/07/15 08:31:20 08:29:08 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 08:31:20 08:28:38 INFO Pod/maas-api-5db58d77cb-dqzbp: Started - Started container maas-api 2026/07/15 08:31:20 08:28:38 INFO Pod/maas-api-5db58d77cb-dqzbp: Created - Created container: maas-api 2026/07/15 08:31:20 08:28:38 INFO Pod/maas-api-5db58d77cb-dqzbp: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-api-rhel9@sha256:435d0a5a849ac8f7c05b7532b53158b29b2bead77295f2da203311777a0f0670" in 755ms (755ms including waiting). Image size: 174970750 bytes. 2026/07/15 08:31:20 08:28:37 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 08:31:20 08:28:37 INFO Pod/maas-api-5db58d77cb-dqzbp: Scheduled - Successfully assigned redhat-ods-applications/maas-api-5db58d77cb-dqzbp to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 08:31:20 08:28:37 INFO Pod/maas-api-5db58d77cb-dqzbp: AddedInterface - Add eth0 [10.129.1.94/23] from ovn-kubernetes 2026/07/15 08:31:20 08:28:37 INFO Deployment/maas-api: ScalingReplicaSet - Scaled up replica set maas-api-5db58d77cb from 0 to 1 2026/07/15 08:31:20 08:28:37 INFO Pod/maas-api-5db58d77cb-dqzbp: Pulling - Pulling image "quay.io/rhoai/odh-maas-api-rhel9@sha256:435d0a5a849ac8f7c05b7532b53158b29b2bead77295f2da203311777a0f0670" 2026/07/15 08:31:20 08:28:37 WARN ReplicaSet/maas-api-5db58d77cb: FailedCreate - Error creating: pods "maas-api-5db58d77cb-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 2026/07/15 08:31:20 08:28:37 INFO ReplicaSet/maas-api-5db58d77cb: SuccessfulCreate - Created pod: maas-api-5db58d77cb-dqzbp 2026/07/15 08:31:20 08:28:19 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-78dcbbdc6c-6ng4w_6ea3276f-b85a-4b5b-9d03-b57c1e0a55cf became leader 2026/07/15 08:31:20 08:28:18 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-7747874bf-ffgt6_8524eb1a-b8e0-4fff-96ec-96104955a9f3 became leader 2026/07/15 08:31:20 08:28:06 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-5b978c765f-7zxff_beb97c06-b0cb-4eec-82f6-8f7f33007556 became leader 2026/07/15 08:31:20 08:28:06 INFO Lease/maas-controller.models-as-a-service.opendatahub.io: LeaderElection - maas-controller-85c784d7b6-dzmw2_94a4c2de-d704-4b33-a659-98f7fa6852e0 became leader 2026/07/15 08:31:20 08:28:02 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-6ng4w: Started - Started container manager 2026/07/15 08:31:20 08:28:02 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-6ng4w: Created - Created container: manager 2026/07/15 08:31:20 08:28:02 INFO Pod/kserve-controller-manager-7747874bf-ffgt6: Created - Created container: manager 2026/07/15 08:31:20 08:28:02 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-6ng4w: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" in 516ms (516ms including waiting). Image size: 235181729 bytes. 2026/07/15 08:31:20 08:28:02 INFO Pod/kserve-controller-manager-7747874bf-ffgt6: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" in 511ms (511ms including waiting). Image size: 237680312 bytes. 2026/07/15 08:31:20 08:28:02 INFO Pod/kserve-controller-manager-7747874bf-ffgt6: Started - Started container manager 2026/07/15 08:31:20 08:28:02 INFO Pod/kserve-controller-manager-7747874bf-ffgt6: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" 2026/07/15 08:31:20 08:28:01 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-6ng4w: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-78dcbbdc6c-6ng4w to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 08:31:20 08:28:01 INFO Pod/kserve-controller-manager-7747874bf-ffgt6: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-7747874bf-ffgt6 to ip-10-0-1-23.us-east-2.compute.internal 2026/07/15 08:31:20 08:28:01 INFO ReplicaSet/kserve-controller-manager-7747874bf: SuccessfulCreate - Created pod: kserve-controller-manager-7747874bf-ffgt6 2026/07/15 08:31:20 08:28:01 INFO Pod/kserve-controller-manager-7747874bf-ffgt6: AddedInterface - Add eth0 [10.130.1.93/23] from ovn-kubernetes 2026/07/15 08:31:20 08:28:01 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-6ng4w: AddedInterface - Add eth0 [10.129.1.93/23] from ovn-kubernetes 2026/07/15 08:31:20 08:28:01 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7747874bf from 0 to 1 2026/07/15 08:31:20 08:28:01 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-78dcbbdc6c from 0 to 1 2026/07/15 08:31:20 08:28:01 INFO ReplicaSet/llmisvc-controller-manager-78dcbbdc6c: SuccessfulCreate - Created pod: llmisvc-controller-manager-78dcbbdc6c-6ng4w 2026/07/15 08:31:20 08:28:01 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-6ng4w: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" 2026/07/15 08:31:20 08:27:51 INFO Pod/model-serving-api-67c44bccdf-g95ks: Started - Started container server 2026/07/15 08:31:20 08:27:51 INFO Pod/model-serving-api-67c44bccdf-g95ks: Created - Created container: server 2026/07/15 08:31:20 08:27:51 INFO Pod/model-serving-api-67c44bccdf-g95ks: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:226370d82da697958a0cd64b5cfa82f290ae1bb701e74b72739fcac0b38a834d" already present on machine 2026/07/15 08:31:20 08:27:50 INFO Pod/model-serving-api-67c44bccdf-g95ks: AddedInterface - Add eth0 [10.129.1.92/23] from ovn-kubernetes 2026/07/15 08:31:20 08:27:50 WARN Pod/model-serving-api-67c44bccdf-g95ks: FailedMount - MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 2026/07/15 08:31:20 08:27:49 INFO Pod/model-serving-api-67c44bccdf-g95ks: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-67c44bccdf-g95ks to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 08:31:20 08:27:49 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 08:31:20 08:27:49 WARN ReplicaSet/model-serving-api-67c44bccdf: FailedCreate - Error creating: pods "model-serving-api-67c44bccdf-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/15 08:31:20 08:27:49 INFO Pod/odh-model-controller-5b978c765f-7zxff: Started - Started container manager 2026/07/15 08:31:20 08:27:49 INFO Pod/odh-model-controller-5b978c765f-7zxff: Created - Created container: manager 2026/07/15 08:31:20 08:27:49 INFO Pod/odh-model-controller-5b978c765f-7zxff: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" in 569ms (569ms including waiting). Image size: 238719886 bytes. 2026/07/15 08:31:20 08:27:49 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-67c44bccdf from 0 to 1 2026/07/15 08:31:20 08:27:49 INFO ReplicaSet/model-serving-api-67c44bccdf: SuccessfulCreate - Created pod: model-serving-api-67c44bccdf-g95ks 2026/07/15 08:31:20 08:27:49 INFO Pod/maas-controller-85c784d7b6-dzmw2: Created - Created container: manager 2026/07/15 08:31:20 08:27:49 INFO Pod/maas-controller-85c784d7b6-dzmw2: Started - Started container manager 2026/07/15 08:31:20 08:27:48 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 08:31:20 08:27:48 INFO Pod/maas-controller-85c784d7b6-dzmw2: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:5effee51faa067115d737a4aa57473635576fb8232b4ea3cc39f76c0146abf65" 2026/07/15 08:31:20 08:27:48 WARN Pod/odh-model-controller-5b978c765f-7zxff: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/15 08:31:20 08:27:48 INFO Pod/maas-controller-85c784d7b6-dzmw2: AddedInterface - Add eth0 [10.129.1.90/23] from ovn-kubernetes 2026/07/15 08:31:20 08:27:48 INFO Pod/maas-controller-85c784d7b6-dzmw2: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:5effee51faa067115d737a4aa57473635576fb8232b4ea3cc39f76c0146abf65" in 626ms (626ms including waiting). Image size: 212346876 bytes. 2026/07/15 08:31:20 08:27:48 INFO Pod/odh-model-controller-5b978c765f-7zxff: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" 2026/07/15 08:31:20 08:27:48 INFO Pod/odh-model-controller-5b978c765f-7zxff: AddedInterface - Add eth0 [10.129.1.91/23] from ovn-kubernetes 2026/07/15 08:31:20 08:27:47 INFO Pod/odh-model-controller-5b978c765f-7zxff: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-5b978c765f-7zxff to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 08:31:20 08:27:47 INFO Pod/maas-controller-85c784d7b6-dzmw2: Scheduled - Successfully assigned redhat-ods-applications/maas-controller-85c784d7b6-dzmw2 to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 08:31:20 08:27:47 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-5b978c765f from 0 to 1 2026/07/15 08:31:20 08:27:47 INFO ReplicaSet/odh-model-controller-5b978c765f: SuccessfulCreate - Created pod: odh-model-controller-5b978c765f-7zxff 2026/07/15 08:31:20 08:27:47 INFO Deployment/maas-controller: ScalingReplicaSet - Scaled up replica set maas-controller-85c784d7b6 from 0 to 1 2026/07/15 08:31:20 08:27:47 INFO ReplicaSet/maas-controller-85c784d7b6: SuccessfulCreate - Created pod: maas-controller-85c784d7b6-dzmw2 2026/07/15 08:31:20 08:27:37 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-s784s: Killing - Stopping container manager 2026/07/15 08:31:20 08:27:32 WARN Pod/llmisvc-controller-manager-78dcbbdc6c-tpj7c: Unhealthy - Readiness probe failed: Get "http://10.130.1.90:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026/07/15 08:31:20 08:27:28 INFO Pod/model-serving-api-67c44bccdf-rxm59: Killing - Stopping container server 2026/07/15 08:31:20 08:27:28 INFO Pod/maas-controller-85c784d7b6-wt9bs: Killing - Stopping container manager 2026/07/15 08:31:20 08:27:28 INFO Pod/maas-api-5db58d77cb-9758b: Killing - Stopping container maas-api 2026/07/15 08:31:20 08:27:28 INFO Pod/odh-model-controller-5b978c765f-8hjvq: Killing - Stopping container manager 2026/07/15 08:31:20 08:27:28 INFO Lease/b7e9931f.trustyai.opendatahub.io: LeaderElection - trustyai-service-operator-controller-manager-bc99cf85b-s784s_4c2aa1e5-a7d1-4d8e-8a40-793f52d0a9ca became leader 2026/07/15 08:31:20 08:27:27 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 08:31:20 08:27:27 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-tpj7c: Killing - Stopping container manager 2026/07/15 08:31:20 08:27:27 INFO Pod/kserve-controller-manager-7747874bf-g58pl: Killing - Stopping container manager 2026/07/15 08:31:20 08:27:11 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-s784s: Scheduled - Successfully assigned redhat-ods-applications/trustyai-service-operator-controller-manager-bc99cf85b-s784s to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 08:31:20 08:27:11 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-94w6g: Killing - Stopping container manager 2026/07/15 08:31:20 08:27:11 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-s784s: AddedInterface - Add eth0 [10.129.1.89/23] from ovn-kubernetes 2026/07/15 08:31:20 08:27:11 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-s784s: Pulled - Container image "quay.io/rhoai/odh-trustyai-service-operator-rhel9@sha256:780ba93a8a4bf718fe19ad7bb9e4dec7b0ae79d5eb6d3e1e5060c3dc2260a515" already present on machine 2026/07/15 08:31:20 08:27:11 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-s784s: Created - Created container: manager 2026/07/15 08:31:20 08:27:11 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-s784s: Started - Started container manager 2026/07/15 08:31:20 08:27:11 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-wjjmw: Killing - Stopping container manager 2026/07/15 08:31:20 08:27:11 INFO ReplicaSet/trustyai-service-operator-controller-manager-bc99cf85b: SuccessfulCreate - Created pod: trustyai-service-operator-controller-manager-bc99cf85b-s784s 2026/07/15 08:31:20 08:27:11 INFO Deployment/trustyai-service-operator-controller-manager: ScalingReplicaSet - Scaled up replica set trustyai-service-operator-controller-manager-bc99cf85b from 0 to 1 2026/07/15 08:31:20 === RESOURCE QUOTAS === 2026/07/15 08:31:20 No resource quotas found in monitored namespaces 2026/07/15 08:31:20 Classification for components: infrastructure/storage (code=1006, confidence=medium) 2026/07/15 08:31:20 Evidence: event Pod/llmisvc-controller-manager-78dcbbdc6c-cpp9l: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/llmisvc-controller-manager-78dcbbdc6c-cpp9l: FailedMount - MountVolume.SetUp failed for volume \"cert\" : secret \"llmisvc-webhook-server-cert\" not found"],"confidence":"medium"} 2026/07/15 08:31:20 Diagnostics complete for failed test: components controller_test.go:483: Stopping: components test failed. 2026/07/15 08:31:20 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/15 08:31:20 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/15 08:31:21 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (1450.77s)