=== RUN TestOdhOperator/components/group_1/kserve/Validate_component_spec testing.go:1811: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_1/kserve/Validate_component_spec (600.01s) === RUN TestOdhOperator/components/group_1/kserve === PAUSE TestOdhOperator/components/group_1/kserve === CONT TestOdhOperator/components/group_1/kserve 2026-07-14T06:44:28Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.1"}, "Cluster": {"type":"OpenShift","version":"4.21.24"}} test_context_test.go:223: Timed out after 600.002s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:114 with: Expected resource 'default-kserve' of kind 'Kserve' to match condition {.spec.nim.managementState == "" []}. Expected <string>: map[apiVersion:components.platform.opendatahub.io/v1alpha1 kind:Kserve metadata:map[annotations:map[component.opendatahub.io/management-state:Managed platform.opendatahub.io/instance.generation:36 platform.opendatahub.io/instance.name:default-dsc platform.opendatahub.io/instance.uid:b1fb39b5-b45f-442c-b1fc-1f88d65747e9 platform.opendatahub.io/type:OpenShift AI Self-Managed platform.opendatahub.io/version:3.5.0-ea.1] creationTimestamp:2026-07-14T06:44:38Z generation:1 labels:map[platform.opendatahub.io/part-of:datasciencecluster] managedFields:[map[apiVersion:components.platform.opendatahub.io/v1alpha1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:component.opendatahub.io/management-state:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:labels:map[f:platform.opendatahub.io/part-of:map[]] f:ownerReferences:map[k:{"uid":"b1fb39b5-b45f-442c-b1fc-1f88d65747e9"}:map[]]] f:spec:map[f:modelsAsService:map[] f:nim:map[] f:rawDeploymentServiceConfig:map[] f:wva:map[]]] manager:datasciencecluster operation:Apply time:2026-07-14T06:47:28Z] map[apiVersion:components.platform.opendatahub.io/v1alpha1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:component.opendatahub.io/management-state:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:labels:map[f:platform.opendatahub.io/part-of:map[]] f:ownerReferences:map[k:{"uid":"b1fb39b5-b45f-442c-b1fc-1f88d65747e9"}:map[]]] f:status:map[f:conditions:map[] f:observedGeneration:map[] f:phase:map[] f:releases:map[k:{"name":"KServe"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-inference-scheduler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-workload-variant-autoscaler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"vLLM"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]]] manager:kserve operation:Apply subresource:status time:2026-07-14T06:47:30Z]] name:default-kserve ownerReferences:[map[apiVersion:datasciencecluster.opendatahub.io/v2 blockOwnerDeletion:true controller:true kind:DataScienceCluster name:default-dsc uid:b1fb39b5-b45f-442c-b1fc-1f88d65747e9]] resourceVersion:102567 uid:0136d072-bece-48df-b66a-7810da13470e] spec:map[modelsAsService:map[managementState:Removed] nim:map[airGapped:false managementState:Managed] rawDeploymentServiceConfig:Headless wva:map[managementState:Removed]] status:map[conditions:[map[lastTransitionTime:2026-07-14T06:47:28Z status:True type:Ready] map[lastTransitionTime:2026-07-14T06:47:30Z observedGeneration:1 status:True type:ProvisioningSucceeded] map[lastTransitionTime:2026-07-14T06:47:30Z observedGeneration:1 status:True type:DeploymentsAvailable] map[lastTransitionTime:2026-07-14T06:47:28Z status:True type:DependenciesAvailable] map[lastTransitionTime:2026-07-14T06:47:28Z status:True type:KserveLLMInferenceServiceDependencies] map[lastTransitionTime:2026-07-14T06:47:29Z status:True type:KserveLLMInferenceServiceWideEPDependencies]] observedGeneration:1 phase:Ready releases:[map[name:KServe repoUrl:https://github.com/kserve/kserve/ version:v0.17.0] map[name:vLLM repoUrl:https://github.com/vllm-project/vllm version:v0.18.0] map[name:llm-d-inference-scheduler repoUrl:https://github.com/llm-d/llm-d-inference-scheduler version:v0.7.1] map[name:llm-d-workload-variant-autoscaler repoUrl:https://github.com/llm-d/llm-d-workload-variant-autoscaler version:v0.6.0]]]] to match expression <string>: .spec.nim.managementState == "" --- FAIL: TestOdhOperator/components/group_1/kserve (670.51s) === RUN TestOdhOperator/components/group_1 --- FAIL: TestOdhOperator/components/group_1 (0.00s) === RUN TestOdhOperator/components 2026/07/14 06:55:38 TEST FAILURE DETECTED: group 1 2026/07/14 06:55:38 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 1) === 2026/07/14 06:55:38 === CLUSTER STATE === 2026/07/14 06:55:38 Node: ip-10-0-132-177.ec2.internal 2026/07/14 06:55:38 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/14 06:55:38 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/14 06:55:38 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/14 06:55:38 Condition Ready: True - kubelet is posting ready status 2026/07/14 06:55:38 Resources: 7500m CPU, 30892156Ki memory allocatable (of 8000m CPU, 32043132Ki memory capacity) 2026/07/14 06:55:38 Node: ip-10-0-137-202.ec2.internal 2026/07/14 06:55:38 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/14 06:55:38 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/14 06:55:38 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/14 06:55:38 Condition Ready: True - kubelet is posting ready status 2026/07/14 06:55:38 Resources: 7500m CPU, 31236228Ki memory allocatable (of 8000m CPU, 32387204Ki memory capacity) 2026/07/14 06:55:38 Node: ip-10-0-137-97.ec2.internal 2026/07/14 06:55:38 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/14 06:55:38 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/14 06:55:38 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/14 06:55:38 Condition Ready: True - kubelet is posting ready status 2026/07/14 06:55:38 Resources: 7500m CPU, 30892168Ki memory allocatable (of 8000m CPU, 32043144Ki memory capacity) 2026/07/14 06:55:38 === DEPLOYMENTS === 2026/07/14 06:55:38 Namespace: redhat-ods-applications 2026/07/14 06:55:38 Deployment dashboard-redirect: 2/2 (READY) 2026/07/14 06:55:38 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/14 06:55:38 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/14 06:55:38 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/14 06:55:38 Deployment model-serving-api: 1/1 (READY) 2026/07/14 06:55:38 Deployment odh-model-controller: 1/1 (READY) 2026/07/14 06:55:38 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/14 06:55:38 Namespace: kube-system 2026/07/14 06:55:38 === PODS === 2026/07/14 06:55:38 Namespace kube-system: no problematic pods found 2026/07/14 06:55:38 Namespace redhat-ods-applications: no problematic pods found 2026/07/14 06:55:38 === OPERATOR STATUS === 2026/07/14 06:55:38 Operator deployment: 3/3 ready 2026/07/14 06:55:38 === DSCI STATUS === 2026/07/14 06:55:38 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/14 06:55:38 === DSC STATUS === 2026/07/14 06:55:38 DSC default-dsc: 2026/07/14 06:55:38 Ready: True - 2026/07/14 06:55:38 ProvisioningSucceeded: True - 2026/07/14 06:55:38 ComponentsReady: True - 2026/07/14 06:55:38 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/14 06:55:38 DashboardReady: False - Component ManagementState is set to Removed 2026/07/14 06:55:38 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/14 06:55:38 KserveLLMInferenceServiceDependencies: True - 2026/07/14 06:55:38 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/14 06:55:38 KserveReady: True - 2026/07/14 06:55:38 KueueReady: False - Component ManagementState is set to Removed 2026/07/14 06:55:38 MLflowOperatorReady: True - 2026/07/14 06:55:38 ModelControllerReady: True - 2026/07/14 06:55:38 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/14 06:55:38 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/14 06:55:38 OGXReady: False - Component ManagementState is set to Removed 2026/07/14 06:55:38 RayReady: False - Component ManagementState is set to Removed 2026/07/14 06:55:38 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/14 06:55:38 TrainerReady: False - Component ManagementState is set to Removed 2026/07/14 06:55:38 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/14 06:55:38 TrustyAIReady: True - 2026/07/14 06:55:38 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/14 06:55:38 === RECENT EVENTS === 2026/07/14 06:55:38 No recent events found in monitored namespaces 2026/07/14 06:55:38 === RESOURCE QUOTAS === 2026/07/14 06:55:38 No resource quotas found in monitored namespaces 2026/07/14 06:55:38 Classification for group 1: infrastructure/dsci-unhealthy (code=1009, confidence=medium) 2026/07/14 06:55:38 Evidence: DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"dsci-unhealthy","error_code":1009,"evidence":["DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False"],"confidence":"medium"} 2026/07/14 06:55:38 Diagnostics complete for failed test: group 1 controller_test.go:752: Stopping: group 1 test failed. 2026/07/14 06:55:38 Circuit breaker: failure classified as infrastructure/dsci-unhealthy (code=1009) 2026/07/14 06:55:38 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/14 06:58:59 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (1450.72s) === RUN TestOdhOperator 2026/07/14 06:42:36 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/14 07:06:57 TEST FAILURE DETECTED: components 2026/07/14 07:06:57 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/14 07:06:57 === CLUSTER STATE === 2026/07/14 07:06:57 Node: ip-10-0-132-177.ec2.internal 2026/07/14 07:06:57 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/14 07:06:57 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/14 07:06:57 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/14 07:06:57 Condition Ready: True - kubelet is posting ready status 2026/07/14 07:06:57 Resources: 7500m CPU, 30892156Ki memory allocatable (of 8000m CPU, 32043132Ki memory capacity) 2026/07/14 07:06:57 Node: ip-10-0-137-202.ec2.internal 2026/07/14 07:06:57 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/14 07:06:57 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/14 07:06:57 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/14 07:06:57 Condition Ready: True - kubelet is posting ready status 2026/07/14 07:06:57 Resources: 7500m CPU, 31236228Ki memory allocatable (of 8000m CPU, 32387204Ki memory capacity) 2026/07/14 07:06:57 Node: ip-10-0-137-97.ec2.internal 2026/07/14 07:06:57 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/14 07:06:57 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/14 07:06:57 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/14 07:06:57 Condition Ready: True - kubelet is posting ready status 2026/07/14 07:06:57 Resources: 7500m CPU, 30892168Ki memory allocatable (of 8000m CPU, 32043144Ki memory capacity) 2026/07/14 07:06:57 === DEPLOYMENTS === 2026/07/14 07:06:57 Namespace: redhat-ods-applications 2026/07/14 07:06:57 Deployment dashboard-redirect: 2/2 (READY) 2026/07/14 07:06:57 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/14 07:06:57 Deployment kubeflow-trainer-controller-manager: 1/1 (READY) 2026/07/14 07:06:57 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/14 07:06:57 Deployment maas-postgres: 1/1 (READY) 2026/07/14 07:06:57 Deployment model-serving-api: 1/1 (READY) 2026/07/14 07:06:57 Deployment odh-model-controller: 1/1 (READY) 2026/07/14 07:06:57 Namespace: kube-system 2026/07/14 07:06:57 === PODS === 2026/07/14 07:06:57 Namespace redhat-ods-applications: no problematic pods found 2026/07/14 07:06:57 Namespace kube-system: no problematic pods found 2026/07/14 07:06:57 === OPERATOR STATUS === 2026/07/14 07:06:57 Operator deployment: 3/3 ready 2026/07/14 07:06:57 === DSCI STATUS === 2026/07/14 07:06:57 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/14 07:06:57 === DSC STATUS === 2026/07/14 07:06:57 DSC default-dsc: 2026/07/14 07:06:57 Ready: True - 2026/07/14 07:06:57 ProvisioningSucceeded: True - 2026/07/14 07:06:57 ComponentsReady: True - 2026/07/14 07:06:57 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/14 07:06:57 DashboardReady: False - Component ManagementState is set to Removed 2026/07/14 07:06:57 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/14 07:06:57 KserveLLMInferenceServiceDependencies: True - 2026/07/14 07:06:57 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/14 07:06:57 KserveReady: True - 2026/07/14 07:06:57 KueueReady: False - Component ManagementState is set to Removed 2026/07/14 07:06:57 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/14 07:06:57 ModelControllerReady: True - 2026/07/14 07:06:57 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/14 07:06:57 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/14 07:06:57 OGXReady: False - Component ManagementState is set to Removed 2026/07/14 07:06:57 RayReady: False - Component ManagementState is set to Removed 2026/07/14 07:06:57 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/14 07:06:57 TrainerReady: True - 2026/07/14 07:06:57 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/14 07:06:57 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/14 07:06:57 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/14 07:06:57 === RECENT EVENTS === 2026/07/14 07:06:57 07:06:39 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-78dcbbdc6c-xg6wn_4b8fa0d8-40ab-42d1-b94a-0485fe7b1ccd became leader 2026/07/14 07:06:57 07:06:36 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-7747874bf-ddhfg_3255babf-d474-496d-8d52-783ea04a6510 became leader 2026/07/14 07:06:57 07:06:27 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-5b978c765f-sfhnf_382d6bc1-3fd9-42fb-86e8-b0ae6db0b014 became leader 2026/07/14 07:06:57 07:06:21 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-xg6wn: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" in 483ms (483ms including waiting). Image size: 235181729 bytes. 2026/07/14 07:06:57 07:06:21 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-xg6wn: Started - Started container manager 2026/07/14 07:06:57 07:06:21 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-xg6wn: Created - Created container: manager 2026/07/14 07:06:57 07:06:20 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-xg6wn: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-78dcbbdc6c-xg6wn to ip-10-0-137-202.ec2.internal 2026/07/14 07:06:57 07:06:20 INFO Pod/kserve-controller-manager-7747874bf-ddhfg: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-7747874bf-ddhfg to ip-10-0-137-97.ec2.internal 2026/07/14 07:06:57 07:06:20 INFO Pod/kserve-controller-manager-7747874bf-ddhfg: Started - Started container manager 2026/07/14 07:06:57 07:06:20 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-78dcbbdc6c from 0 to 1 2026/07/14 07:06:57 07:06:20 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-xg6wn: AddedInterface - Add eth0 [10.133.0.64/23] from ovn-kubernetes 2026/07/14 07:06:57 07:06:20 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-xg6wn: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" 2026/07/14 07:06:57 07:06:20 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7747874bf from 0 to 1 2026/07/14 07:06:57 07:06:20 INFO Pod/kserve-controller-manager-7747874bf-ddhfg: Created - Created container: manager 2026/07/14 07:06:57 07:06:20 INFO Pod/kserve-controller-manager-7747874bf-ddhfg: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" in 348ms (348ms including waiting). Image size: 237680312 bytes. 2026/07/14 07:06:57 07:06:20 INFO Pod/kserve-controller-manager-7747874bf-ddhfg: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" 2026/07/14 07:06:57 07:06:20 INFO Pod/kserve-controller-manager-7747874bf-ddhfg: AddedInterface - Add eth0 [10.132.0.120/23] from ovn-kubernetes 2026/07/14 07:06:57 07:06:20 INFO ReplicaSet/llmisvc-controller-manager-78dcbbdc6c: SuccessfulCreate - Created pod: llmisvc-controller-manager-78dcbbdc6c-xg6wn 2026/07/14 07:06:57 07:06:20 INFO ReplicaSet/kserve-controller-manager-7747874bf: SuccessfulCreate - Created pod: kserve-controller-manager-7747874bf-ddhfg 2026/07/14 07:06:57 07:06:10 INFO Pod/model-serving-api-67c44bccdf-7l25q: Started - Started container server 2026/07/14 07:06:57 07:06:10 INFO Pod/model-serving-api-67c44bccdf-7l25q: AddedInterface - Add eth0 [10.132.0.119/23] from ovn-kubernetes 2026/07/14 07:06:57 07:06:10 INFO Pod/model-serving-api-67c44bccdf-7l25q: Created - Created container: server 2026/07/14 07:06:57 07:06:10 INFO Pod/model-serving-api-67c44bccdf-7l25q: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:226370d82da697958a0cd64b5cfa82f290ae1bb701e74b72739fcac0b38a834d" already present on machine 2026/07/14 07:06:57 07:06:09 INFO Pod/model-serving-api-67c44bccdf-7l25q: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-67c44bccdf-7l25q to ip-10-0-137-97.ec2.internal 2026/07/14 07:06:57 07:06:09 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/14 07:06:57 07:06:09 INFO Pod/odh-model-controller-5b978c765f-sfhnf: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" 2026/07/14 07:06:57 07:06:09 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-67c44bccdf from 0 to 1 2026/07/14 07:06:57 07:06:09 WARN ReplicaSet/model-serving-api-67c44bccdf: FailedCreate - Error creating: pods "model-serving-api-67c44bccdf-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/14 07:06:57 07:06:09 WARN Pod/model-serving-api-67c44bccdf-7l25q: FailedMount - MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 2026/07/14 07:06:57 07:06:09 INFO Pod/odh-model-controller-5b978c765f-sfhnf: AddedInterface - Add eth0 [10.132.0.118/23] from ovn-kubernetes 2026/07/14 07:06:57 07:06:09 INFO Pod/odh-model-controller-5b978c765f-sfhnf: Started - Started container manager 2026/07/14 07:06:57 07:06:09 INFO Pod/odh-model-controller-5b978c765f-sfhnf: Created - Created container: manager 2026/07/14 07:06:57 07:06:09 INFO Pod/odh-model-controller-5b978c765f-sfhnf: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" in 525ms (525ms including waiting). Image size: 238719886 bytes. 2026/07/14 07:06:57 07:06:09 INFO ReplicaSet/model-serving-api-67c44bccdf: SuccessfulCreate - Created pod: model-serving-api-67c44bccdf-7l25q 2026/07/14 07:06:57 07:06:08 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/14 07:06:57 07:06:08 INFO Pod/odh-model-controller-5b978c765f-sfhnf: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-5b978c765f-sfhnf to ip-10-0-137-97.ec2.internal 2026/07/14 07:06:57 07:06:08 INFO ReplicaSet/odh-model-controller-5b978c765f: SuccessfulCreate - Created pod: odh-model-controller-5b978c765f-sfhnf 2026/07/14 07:06:57 07:06:08 WARN Pod/odh-model-controller-5b978c765f-sfhnf: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/14 07:06:57 07:06:08 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-5b978c765f from 0 to 1 2026/07/14 07:06:57 07:06:03 WARN Pod/kserve-controller-manager-7747874bf-sgtsc: Unhealthy - Readiness probe failed: Get "http://10.132.0.117:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026/07/14 07:06:57 07:05:58 INFO Pod/model-serving-api-67c44bccdf-th2b6: Killing - Stopping container server 2026/07/14 07:06:57 07:05:58 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-w2njk: Killing - Stopping container manager 2026/07/14 07:06:57 07:05:58 INFO Pod/odh-model-controller-5b978c765f-n2wkw: Killing - Stopping container manager 2026/07/14 07:06:57 07:05:58 INFO Pod/kserve-controller-manager-7747874bf-sgtsc: Killing - Stopping container manager 2026/07/14 07:06:57 07:05:30 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-78dcbbdc6c-w2njk_720aa9e8-e358-411f-b284-8d06e9918930 became leader 2026/07/14 07:06:57 07:05:27 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-7747874bf-sgtsc_12b8feb0-b683-45fb-a25c-d0215ff3185b became leader 2026/07/14 07:06:57 07:05:14 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-5b978c765f-n2wkw_894993fd-a4bd-492a-9927-7ab75a6b00b5 became leader 2026/07/14 07:06:57 07:05:13 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-w2njk: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" in 2.7s (2.7s including waiting). Image size: 235181729 bytes. 2026/07/14 07:06:57 07:05:13 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-w2njk: Started - Started container manager 2026/07/14 07:06:57 07:05:13 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-w2njk: Created - Created container: manager 2026/07/14 07:06:57 07:05:11 INFO Pod/kserve-controller-manager-7747874bf-sgtsc: Created - Created container: manager 2026/07/14 07:06:57 07:05:11 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-7dbc595499-9dx5d_cfaf435e-0665-4916-be9b-7abee4e33d50 became leader 2026/07/14 07:06:57 07:05:11 INFO Pod/kserve-controller-manager-7747874bf-sgtsc: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" in 836ms (836ms including waiting). Image size: 237680312 bytes. 2026/07/14 07:06:57 07:05:11 INFO Pod/kserve-controller-manager-7747874bf-sgtsc: Started - Started container manager 2026/07/14 07:06:57 07:05:11 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-w2njk: AddedInterface - Add eth0 [10.133.0.63/23] from ovn-kubernetes 2026/07/14 07:06:57 07:05:11 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-w2njk: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" 2026/07/14 07:06:57 07:05:10 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-w2njk: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-78dcbbdc6c-w2njk to ip-10-0-137-202.ec2.internal 2026/07/14 07:06:57 07:05:10 INFO Pod/kserve-controller-manager-7747874bf-sgtsc: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" 2026/07/14 07:06:57 07:05:10 WARN Pod/llmisvc-controller-manager-78dcbbdc6c-w2njk: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 2026/07/14 07:06:57 07:05:10 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-78dcbbdc6c from 0 to 1 2026/07/14 07:06:57 07:05:10 INFO ReplicaSet/llmisvc-controller-manager-78dcbbdc6c: SuccessfulCreate - Created pod: llmisvc-controller-manager-78dcbbdc6c-w2njk 2026/07/14 07:06:57 07:05:10 INFO Pod/kserve-controller-manager-7747874bf-sgtsc: AddedInterface - Add eth0 [10.132.0.117/23] from ovn-kubernetes 2026/07/14 07:06:57 07:05:09 INFO Pod/kserve-controller-manager-7747874bf-sgtsc: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-7747874bf-sgtsc to ip-10-0-137-97.ec2.internal 2026/07/14 07:06:57 07:05:09 INFO ReplicaSet/kserve-controller-manager-7747874bf: SuccessfulCreate - Created pod: kserve-controller-manager-7747874bf-sgtsc 2026/07/14 07:06:57 07:05:09 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7747874bf from 0 to 1 2026/07/14 07:06:57 07:04:59 INFO Pod/model-serving-api-67c44bccdf-th2b6: AddedInterface - Add eth0 [10.132.0.116/23] from ovn-kubernetes 2026/07/14 07:06:57 07:04:59 INFO Pod/odh-model-controller-5b978c765f-n2wkw: Started - Started container manager 2026/07/14 07:06:57 07:04:59 INFO Pod/model-serving-api-67c44bccdf-th2b6: Created - Created container: server 2026/07/14 07:06:57 07:04:59 INFO Pod/odh-model-controller-5b978c765f-n2wkw: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" in 468ms (468ms including waiting). Image size: 238719886 bytes. 2026/07/14 07:06:57 07:04:59 INFO Pod/model-serving-api-67c44bccdf-th2b6: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:226370d82da697958a0cd64b5cfa82f290ae1bb701e74b72739fcac0b38a834d" already present on machine 2026/07/14 07:06:57 07:04:59 INFO Pod/odh-model-controller-5b978c765f-n2wkw: Created - Created container: manager 2026/07/14 07:06:57 07:04:59 INFO Pod/model-serving-api-67c44bccdf-th2b6: Started - Started container server 2026/07/14 07:06:57 07:04:58 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/14 07:06:57 07:04:58 INFO Pod/model-serving-api-67c44bccdf-th2b6: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-67c44bccdf-th2b6 to ip-10-0-137-97.ec2.internal 2026/07/14 07:06:57 07:04:58 WARN Pod/odh-model-controller-5b978c765f-n2wkw: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/14 07:06:57 07:04:58 INFO Pod/odh-model-controller-5b978c765f-n2wkw: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" 2026/07/14 07:06:57 07:04:58 INFO ReplicaSet/model-serving-api-67c44bccdf: SuccessfulCreate - Created pod: model-serving-api-67c44bccdf-th2b6 2026/07/14 07:06:57 07:04:58 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-67c44bccdf from 0 to 1 2026/07/14 07:06:57 07:04:58 INFO Pod/odh-model-controller-5b978c765f-n2wkw: AddedInterface - Add eth0 [10.132.0.115/23] from ovn-kubernetes 2026/07/14 07:06:57 07:04:58 WARN ReplicaSet/model-serving-api-67c44bccdf: FailedCreate - Error creating: pods "model-serving-api-67c44bccdf-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/14 07:06:57 07:04:57 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/14 07:06:57 07:04:57 INFO Pod/odh-model-controller-5b978c765f-n2wkw: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-5b978c765f-n2wkw to ip-10-0-137-97.ec2.internal 2026/07/14 07:06:57 07:04:57 INFO ReplicaSet/odh-model-controller-5b978c765f: SuccessfulCreate - Created pod: odh-model-controller-5b978c765f-n2wkw 2026/07/14 07:06:57 07:04:57 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-5b978c765f from 0 to 1 2026/07/14 07:06:57 07:04:53 INFO Pod/kubeflow-trainer-controller-manager-7dbc595499-9dx5d: Started - Started container manager 2026/07/14 07:06:57 07:04:53 INFO Pod/kubeflow-trainer-controller-manager-7dbc595499-9dx5d: AddedInterface - Add eth0 [10.132.0.114/23] from ovn-kubernetes 2026/07/14 07:06:57 07:04:53 INFO Pod/kubeflow-trainer-controller-manager-7dbc595499-9dx5d: Pulled - Container image "quay.io/rhoai/odh-trainer-rhel9@sha256:595f456a6b3900dd112fa04416a2d91f448f8d68031b0c2de0d1dcb5fed28aa1" already present on machine 2026/07/14 07:06:57 07:04:53 INFO Pod/kubeflow-trainer-controller-manager-7dbc595499-9dx5d: Created - Created container: manager 2026/07/14 07:06:57 07:04:52 INFO Pod/kubeflow-trainer-controller-manager-7dbc595499-9dx5d: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-7dbc595499-9dx5d to ip-10-0-137-97.ec2.internal 2026/07/14 07:06:57 07:04:52 WARN ReplicaSet/llmisvc-controller-manager-78dcbbdc6c: FailedCreate - Error creating: pods "llmisvc-controller-manager-78dcbbdc6c-" is forbidden: error looking up service account redhat-ods-applications/llmisvc-controller-manager: serviceaccount "llmisvc-controller-manager" not found 2026/07/14 07:06:57 07:04:52 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-7dbc595499 from 0 to 1 2026/07/14 07:06:57 07:04:52 INFO ReplicaSet/kubeflow-trainer-controller-manager-7dbc595499: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-7dbc595499-9dx5d 2026/07/14 07:06:57 07:04:52 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7747874bf from 0 to 1 2026/07/14 07:06:57 07:04:52 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-78dcbbdc6c from 0 to 1 2026/07/14 07:06:57 07:04:52 WARN ReplicaSet/kserve-controller-manager-7747874bf: FailedCreate - Error creating: pods "kserve-controller-manager-7747874bf-" is forbidden: error looking up service account redhat-ods-applications/kserve-controller-manager: serviceaccount "kserve-controller-manager" not found 2026/07/14 07:06:57 07:04:47 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-p5vgx: Killing - Stopping container manager 2026/07/14 07:06:57 07:04:47 INFO Pod/odh-model-controller-5b978c765f-bs8sh: Killing - Stopping container manager 2026/07/14 07:06:57 07:04:47 INFO Pod/kserve-controller-manager-7747874bf-zrjdc: Killing - Stopping container manager 2026/07/14 07:06:57 07:04:47 INFO Pod/model-serving-api-67c44bccdf-tltpz: Killing - Stopping container server 2026/07/14 07:06:57 07:04:36 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/14 07:06:57 07:04:36 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/14 07:06:57 07:04:36 INFO Pod/maas-api-5588fcb4cd-tljvq: Killing - Stopping container maas-api 2026/07/14 07:06:57 07:04:36 INFO Pod/maas-controller-85c784d7b6-zvgmh: Killing - Stopping container manager 2026/07/14 07:06:57 07:04:12 INFO Pod/maas-api-5588fcb4cd-tljvq: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-api-rhel9@sha256:435d0a5a849ac8f7c05b7532b53158b29b2bead77295f2da203311777a0f0670" in 1.467s (1.467s including waiting). Image size: 174970750 bytes. 2026/07/14 07:06:57 07:04:12 INFO Pod/maas-api-5588fcb4cd-tljvq: Created - Created container: maas-api 2026/07/14 07:06:57 07:04:12 INFO Pod/maas-api-5588fcb4cd-tljvq: Started - Started container maas-api 2026/07/14 07:06:57 07:04:10 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/14 07:06:57 07:04:10 INFO Pod/maas-api-5588fcb4cd-tljvq: Scheduled - Successfully assigned redhat-ods-applications/maas-api-5588fcb4cd-tljvq to ip-10-0-137-202.ec2.internal 2026/07/14 07:06:57 07:04:10 INFO Deployment/maas-api: ScalingReplicaSet - Scaled up replica set maas-api-5588fcb4cd from 0 to 1 2026/07/14 07:06:57 07:04:10 INFO Pod/maas-api-5588fcb4cd-tljvq: AddedInterface - Add eth0 [10.133.0.61/23] from ovn-kubernetes 2026/07/14 07:06:57 07:04:10 WARN ReplicaSet/maas-api-5588fcb4cd: FailedCreate - Error creating: pods "maas-api-5588fcb4cd-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 2026/07/14 07:06:57 07:04:10 INFO ReplicaSet/maas-api-5588fcb4cd: SuccessfulCreate - Created pod: maas-api-5588fcb4cd-tljvq 2026/07/14 07:06:57 07:04:10 INFO Pod/maas-api-5588fcb4cd-tljvq: Pulling - Pulling image "quay.io/rhoai/odh-maas-api-rhel9@sha256:435d0a5a849ac8f7c05b7532b53158b29b2bead77295f2da203311777a0f0670" 2026/07/14 07:06:57 07:03:39 INFO Lease/maas-controller.models-as-a-service.opendatahub.io: LeaderElection - maas-controller-85c784d7b6-zvgmh_a5c4b1d4-c457-491c-82ca-011a121d7b2d became leader 2026/07/14 07:06:57 07:03:39 INFO Pod/maas-controller-85c784d7b6-zvgmh: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:5effee51faa067115d737a4aa57473635576fb8232b4ea3cc39f76c0146abf65" in 2.596s (2.596s including waiting). Image size: 212346876 bytes. 2026/07/14 07:06:57 07:03:39 INFO Pod/maas-controller-85c784d7b6-zvgmh: Created - Created container: manager 2026/07/14 07:06:57 07:03:39 INFO Pod/maas-controller-85c784d7b6-zvgmh: Started - Started container manager 2026/07/14 07:06:57 07:03:37 INFO Pod/maas-controller-85c784d7b6-zvgmh: AddedInterface - Add eth0 [10.133.0.60/23] from ovn-kubernetes 2026/07/14 07:06:57 07:03:37 INFO Pod/maas-controller-85c784d7b6-zvgmh: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:5effee51faa067115d737a4aa57473635576fb8232b4ea3cc39f76c0146abf65" 2026/07/14 07:06:57 07:03:36 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/14 07:06:57 07:03:36 INFO Pod/maas-controller-85c784d7b6-zvgmh: Scheduled - Successfully assigned redhat-ods-applications/maas-controller-85c784d7b6-zvgmh to ip-10-0-137-202.ec2.internal 2026/07/14 07:06:57 07:03:36 INFO ReplicaSet/maas-controller-85c784d7b6: SuccessfulCreate - Created pod: maas-controller-85c784d7b6-zvgmh 2026/07/14 07:06:57 07:03:36 INFO Deployment/maas-controller: ScalingReplicaSet - Scaled up replica set maas-controller-85c784d7b6 from 0 to 1 2026/07/14 07:06:57 07:03:19 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-78dcbbdc6c-p5vgx_75c1d349-5b72-40ef-9962-d54f46751f16 became leader 2026/07/14 07:06:57 07:03:17 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-7747874bf-zrjdc_99c4c41d-13e9-4f59-af55-9e75bdbb07d0 became leader 2026/07/14 07:06:57 07:03:07 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-5b978c765f-bs8sh_61dd4f2c-fbff-4f73-b946-f5a22d370213 became leader 2026/07/14 07:06:57 07:03:01 INFO Pod/kserve-controller-manager-7747874bf-zrjdc: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" in 613ms (613ms including waiting). Image size: 237680312 bytes. 2026/07/14 07:06:57 07:03:01 INFO Pod/kserve-controller-manager-7747874bf-zrjdc: Created - Created container: manager 2026/07/14 07:06:57 07:03:01 INFO Pod/kserve-controller-manager-7747874bf-zrjdc: Started - Started container manager 2026/07/14 07:06:57 07:03:00 INFO Pod/kserve-controller-manager-7747874bf-zrjdc: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" 2026/07/14 07:06:57 07:03:00 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-p5vgx: Created - Created container: manager 2026/07/14 07:06:57 07:03:00 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-p5vgx: Started - Started container manager 2026/07/14 07:06:57 07:03:00 INFO Pod/kserve-controller-manager-7747874bf-zrjdc: AddedInterface - Add eth0 [10.132.0.112/23] from ovn-kubernetes 2026/07/14 07:06:57 07:03:00 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-p5vgx: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" in 503ms (503ms including waiting). Image size: 235181729 bytes. 2026/07/14 07:06:57 07:03:00 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-p5vgx: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" 2026/07/14 07:06:57 07:03:00 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-p5vgx: AddedInterface - Add eth0 [10.132.0.113/23] from ovn-kubernetes 2026/07/14 07:06:57 07:02:59 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-p5vgx: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-78dcbbdc6c-p5vgx to ip-10-0-137-97.ec2.internal 2026/07/14 07:06:57 07:02:59 INFO Pod/kserve-controller-manager-7747874bf-zrjdc: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-7747874bf-zrjdc to ip-10-0-137-97.ec2.internal 2026/07/14 07:06:57 07:02:59 INFO ReplicaSet/kserve-controller-manager-7747874bf: SuccessfulCreate - Created pod: kserve-controller-manager-7747874bf-zrjdc 2026/07/14 07:06:57 07:02:59 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-78dcbbdc6c from 0 to 1 2026/07/14 07:06:57 07:02:59 WARN Pod/kserve-controller-manager-7747874bf-zrjdc: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 2026/07/14 07:06:57 07:02:59 INFO ReplicaSet/llmisvc-controller-manager-78dcbbdc6c: SuccessfulCreate - Created pod: llmisvc-controller-manager-78dcbbdc6c-p5vgx 2026/07/14 07:06:57 07:02:59 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7747874bf from 0 to 1 2026/07/14 07:06:57 07:02:49 INFO Pod/odh-model-controller-5b978c765f-bs8sh: Started - Started container manager 2026/07/14 07:06:57 07:02:49 INFO Pod/odh-model-controller-5b978c765f-bs8sh: Created - Created container: manager 2026/07/14 07:06:57 07:02:49 INFO Pod/model-serving-api-67c44bccdf-tltpz: AddedInterface - Add eth0 [10.132.0.111/23] from ovn-kubernetes 2026/07/14 07:06:57 07:02:49 INFO Pod/odh-model-controller-5b978c765f-bs8sh: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" in 1.677s (1.677s including waiting). Image size: 238719886 bytes. 2026/07/14 07:06:57 07:02:49 INFO Pod/model-serving-api-67c44bccdf-tltpz: Created - Created container: server 2026/07/14 07:06:57 07:02:49 INFO Pod/model-serving-api-67c44bccdf-tltpz: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:226370d82da697958a0cd64b5cfa82f290ae1bb701e74b72739fcac0b38a834d" already present on machine 2026/07/14 07:06:57 07:02:49 INFO Pod/model-serving-api-67c44bccdf-tltpz: Started - Started container server 2026/07/14 07:06:57 07:02:48 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/14 07:06:57 07:02:48 INFO Pod/model-serving-api-67c44bccdf-tltpz: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-67c44bccdf-tltpz to ip-10-0-137-97.ec2.internal 2026/07/14 07:06:57 07:02:48 WARN ReplicaSet/model-serving-api-67c44bccdf: FailedCreate - Error creating: pods "model-serving-api-67c44bccdf-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/14 07:06:57 07:02:48 INFO ReplicaSet/model-serving-api-67c44bccdf: SuccessfulCreate - Created pod: model-serving-api-67c44bccdf-tltpz 2026/07/14 07:06:57 07:02:48 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-67c44bccdf from 0 to 1 2026/07/14 07:06:57 07:02:47 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/14 07:06:57 07:02:47 INFO Pod/odh-model-controller-5b978c765f-bs8sh: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-5b978c765f-bs8sh to ip-10-0-137-97.ec2.internal 2026/07/14 07:06:57 07:02:47 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-5b978c765f from 0 to 1 2026/07/14 07:06:57 07:02:47 INFO ReplicaSet/odh-model-controller-5b978c765f: SuccessfulCreate - Created pod: odh-model-controller-5b978c765f-bs8sh 2026/07/14 07:06:57 07:02:47 INFO Pod/odh-model-controller-5b978c765f-bs8sh: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" 2026/07/14 07:06:57 07:02:47 INFO Pod/odh-model-controller-5b978c765f-bs8sh: AddedInterface - Add eth0 [10.132.0.110/23] from ovn-kubernetes 2026/07/14 07:06:57 07:02:38 INFO Pod/maas-postgres-bb9d55498-cd92v: Pulled - Successfully pulled image "registry.redhat.io/rhel9/postgresql-15:latest" in 2.334s (2.334s including waiting). Image size: 400818801 bytes. 2026/07/14 07:06:57 07:02:38 INFO Pod/maas-postgres-bb9d55498-cd92v: Created - Created container: postgres 2026/07/14 07:06:57 07:02:38 INFO Pod/maas-postgres-bb9d55498-cd92v: Started - Started container postgres 2026/07/14 07:06:57 07:02:36 INFO Pod/maas-postgres-bb9d55498-cd92v: AddedInterface - Add eth0 [10.132.0.108/23] from ovn-kubernetes 2026/07/14 07:06:57 07:02:36 INFO Pod/maas-postgres-bb9d55498-cd92v: Pulling - Pulling image "registry.redhat.io/rhel9/postgresql-15:latest" 2026/07/14 07:06:57 07:02:35 INFO Pod/maas-postgres-bb9d55498-cd92v: Scheduled - Successfully assigned redhat-ods-applications/maas-postgres-bb9d55498-cd92v to ip-10-0-137-97.ec2.internal 2026/07/14 07:06:57 07:02:35 INFO ReplicaSet/maas-postgres-bb9d55498: SuccessfulCreate - Created pod: maas-postgres-bb9d55498-cd92v 2026/07/14 07:06:57 07:02:35 INFO Deployment/maas-postgres: ScalingReplicaSet - Scaled up replica set maas-postgres-bb9d55498 from 0 to 1 2026/07/14 07:06:57 07:02:26 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-cpj7s: Killing - Stopping container manager 2026/07/14 07:06:57 07:02:19 INFO Lease/b7e9931f.trustyai.opendatahub.io: LeaderElection - trustyai-service-operator-controller-manager-bc99cf85b-cpj7s_23899196-3eda-4fe0-b673-4003b5a121d8 became leader 2026/07/14 07:06:57 07:02:16 INFO Pod/model-serving-api-67c44bccdf-zrptz: Killing - Stopping container server 2026/07/14 07:06:57 07:02:16 INFO Pod/odh-model-controller-5b978c765f-4lrf4: Killing - Stopping container manager 2026/07/14 07:06:57 07:02:16 INFO Pod/kserve-controller-manager-7747874bf-7ms8m: Killing - Stopping container manager 2026/07/14 07:06:57 07:02:16 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-wzshb: Killing - Stopping container manager 2026/07/14 07:06:57 07:02:01 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-cpj7s: AddedInterface - Add eth0 [10.132.0.107/23] from ovn-kubernetes 2026/07/14 07:06:57 07:02:01 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-cpj7s: Pulled - Container image "quay.io/rhoai/odh-trustyai-service-operator-rhel9@sha256:780ba93a8a4bf718fe19ad7bb9e4dec7b0ae79d5eb6d3e1e5060c3dc2260a515" already present on machine 2026/07/14 07:06:57 07:02:01 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-cpj7s: Created - Created container: manager 2026/07/14 07:06:57 07:02:01 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-cpj7s: Started - Started container manager 2026/07/14 07:06:57 07:02:00 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-cpj7s: Scheduled - Successfully assigned redhat-ods-applications/trustyai-service-operator-controller-manager-bc99cf85b-cpj7s to ip-10-0-137-97.ec2.internal 2026/07/14 07:06:57 07:02:00 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-mw58w: Killing - Stopping container manager 2026/07/14 07:06:57 07:02:00 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-qhqll: Killing - Stopping container manager 2026/07/14 07:06:57 07:02:00 INFO ReplicaSet/trustyai-service-operator-controller-manager-bc99cf85b: SuccessfulCreate - Created pod: trustyai-service-operator-controller-manager-bc99cf85b-cpj7s 2026/07/14 07:06:57 07:02:00 INFO Deployment/trustyai-service-operator-controller-manager: ScalingReplicaSet - Scaled up replica set trustyai-service-operator-controller-manager-bc99cf85b from 0 to 1 2026/07/14 07:06:57 === RESOURCE QUOTAS === 2026/07/14 07:06:57 No resource quotas found in monitored namespaces 2026/07/14 07:06:57 Classification for components: infrastructure/storage (code=1006, confidence=medium) 2026/07/14 07:06:57 Evidence: event Pod/model-serving-api-67c44bccdf-7l25q: FailedMount - MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/model-serving-api-67c44bccdf-7l25q: FailedMount - MountVolume.SetUp failed for volume \"tls-certs\" : secret \"model-serving-api-tls\" not found"],"confidence":"medium"} 2026/07/14 07:06:57 Diagnostics complete for failed test: components controller_test.go:483: Stopping: components test failed. 2026/07/14 07:06:57 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/14 07:06:57 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/14 07:06:57 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (1461.66s)