=== RUN TestOdhOperator/components/group_1/kserve/Validate_component_spec testing.go:1811: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_1/kserve/Validate_component_spec (600.03s) === RUN TestOdhOperator/components/group_1/kserve === PAUSE TestOdhOperator/components/group_1/kserve === CONT TestOdhOperator/components/group_1/kserve 2026-07-15T04:45:17Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.1"}, "Cluster": {"type":"OpenShift","version":"4.21.19"}} test_context_test.go:223: Timed out after 600.008s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:114 with: Expected resource 'default-kserve' of kind 'Kserve' to match condition {.spec.nim.managementState == "" []}. Expected <string>: map[apiVersion:components.platform.opendatahub.io/v1alpha1 kind:Kserve metadata:map[annotations:map[component.opendatahub.io/management-state:Managed platform.opendatahub.io/instance.generation:105 platform.opendatahub.io/instance.name:default-dsc platform.opendatahub.io/instance.uid:9013f03c-d6c8-43c1-bdd4-c3e666f6094f platform.opendatahub.io/type:OpenShift AI Self-Managed platform.opendatahub.io/version:3.5.0-ea.1] creationTimestamp:2026-07-15T04:45:31Z generation:1 labels:map[platform.opendatahub.io/part-of:datasciencecluster] managedFields:[map[apiVersion:components.platform.opendatahub.io/v1alpha1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:component.opendatahub.io/management-state:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:labels:map[f:platform.opendatahub.io/part-of:map[]] f:ownerReferences:map[k:{"uid":"9013f03c-d6c8-43c1-bdd4-c3e666f6094f"}:map[]]] f:spec:map[f:modelsAsService:map[f:managementState:map[]] f:nim:map[] f:rawDeploymentServiceConfig:map[] f:wva:map[f:managementState:map[]]]] manager:datasciencecluster operation:Apply time:2026-07-15T04:51:21Z] map[apiVersion:components.platform.opendatahub.io/v1alpha1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:component.opendatahub.io/management-state:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:labels:map[f:platform.opendatahub.io/part-of:map[]] f:ownerReferences:map[k:{"uid":"9013f03c-d6c8-43c1-bdd4-c3e666f6094f"}:map[]]] f:status:map[f:conditions:map[] f:observedGeneration:map[] f:phase:map[] f:releases:map[k:{"name":"KServe"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-inference-scheduler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-workload-variant-autoscaler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"vLLM"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]]] manager:kserve operation:Apply subresource:status time:2026-07-15T04:56:30Z]] name:default-kserve ownerReferences:[map[apiVersion:datasciencecluster.opendatahub.io/v2 blockOwnerDeletion:true controller:true kind:DataScienceCluster name:default-dsc uid:9013f03c-d6c8-43c1-bdd4-c3e666f6094f]] resourceVersion:33148569 uid:78caa303-36e1-4b50-b76c-9552c683e9a0] spec:map[modelsAsService:map[managementState:Managed] nim:map[airGapped:false managementState:Managed] rawDeploymentServiceConfig:Headless wva:map[managementState:Removed]] status:map[conditions:[map[lastTransitionTime:2026-07-15T04:56:29Z status:True type:Ready] map[lastTransitionTime:2026-07-15T04:56:30Z observedGeneration:1 status:True type:ProvisioningSucceeded] map[lastTransitionTime:2026-07-15T04:56:30Z observedGeneration:1 status:True type:DeploymentsAvailable] map[lastTransitionTime:2026-07-15T04:56:29Z status:True type:DependenciesAvailable] map[lastTransitionTime:2026-07-15T04:56:29Z status:True type:KserveLLMInferenceServiceDependencies] map[lastTransitionTime:2026-07-15T04:56:29Z status:True type:KserveLLMInferenceServiceWideEPDependencies]] observedGeneration:1 phase:Ready releases:[map[name:KServe repoUrl:https://github.com/kserve/kserve/ version:v0.17.0] map[name:vLLM repoUrl:https://github.com/vllm-project/vllm version:v0.18.0] map[name:llm-d-inference-scheduler repoUrl:https://github.com/llm-d/llm-d-inference-scheduler version:v0.7.1] map[name:llm-d-workload-variant-autoscaler repoUrl:https://github.com/llm-d/llm-d-workload-variant-autoscaler version:v0.6.0]]]] to match expression <string>: .spec.nim.managementState == "" --- FAIL: TestOdhOperator/components/group_1/kserve (682.71s) === RUN TestOdhOperator/components/group_1 --- FAIL: TestOdhOperator/components/group_1 (0.00s) === RUN TestOdhOperator/components 2026/07/15 04:56:40 TEST FAILURE DETECTED: group 1 2026/07/15 04:56:40 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 1) === 2026/07/15 04:56:40 === CLUSTER STATE === 2026/07/15 04:56:40 Node: ip-10-0-1-23.us-east-2.compute.internal 2026/07/15 04:56:40 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 04:56:40 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 04:56:40 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 04:56:40 Condition Ready: True - kubelet is posting ready status 2026/07/15 04:56:40 Resources: 15500m CPU, 63182900Ki memory allocatable (of 16000m CPU, 64333876Ki memory capacity) 2026/07/15 04:56:40 Node: ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 04:56:40 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 04:56:40 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 04:56:40 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 04:56:40 Condition Ready: True - kubelet is posting ready status 2026/07/15 04:56:40 Resources: 15500m CPU, 63871012Ki memory allocatable (of 16000m CPU, 65021988Ki memory capacity) 2026/07/15 04:56:40 Node: ip-10-0-1-62.us-east-2.compute.internal 2026/07/15 04:56:40 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 04:56:40 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 04:56:40 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 04:56:40 Condition Ready: True - kubelet is posting ready status 2026/07/15 04:56:40 Resources: 15500m CPU, 63182896Ki memory allocatable (of 16000m CPU, 64333872Ki memory capacity) 2026/07/15 04:56:40 === DEPLOYMENTS === 2026/07/15 04:56:40 Namespace: redhat-ods-applications 2026/07/15 04:56:40 Deployment dashboard-redirect: 2/2 (READY) 2026/07/15 04:56:40 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/15 04:56:40 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/15 04:56:40 Deployment maas-api: 1/1 (READY) 2026/07/15 04:56:40 Deployment maas-controller: 1/1 (READY) 2026/07/15 04:56:40 Deployment maas-postgres: 1/1 (READY) 2026/07/15 04:56:40 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/15 04:56:40 Deployment model-serving-api: 1/1 (READY) 2026/07/15 04:56:40 Deployment odh-model-controller: 1/1 (READY) 2026/07/15 04:56:40 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/15 04:56:40 Namespace: kube-system 2026/07/15 04:56:40 === PODS === 2026/07/15 04:56:40 Failed to collect pod data: redhat-ods-applications/trustyai-service-operator-controller-manager-bc99cf85b-8225f: container manager restarts=2 2026/07/15 04:56:40 === OPERATOR STATUS === 2026/07/15 04:56:40 Operator deployment: 3/3 ready 2026/07/15 04:56:40 === DSCI STATUS === 2026/07/15 04:56:40 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/15 04:56:40 === DSC STATUS === 2026/07/15 04:56:40 DSC default-dsc: 2026/07/15 04:56:40 Ready: True - 2026/07/15 04:56:40 ProvisioningSucceeded: True - 2026/07/15 04:56:40 ComponentsReady: True - 2026/07/15 04:56:40 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/15 04:56:40 DashboardReady: False - Component ManagementState is set to Removed 2026/07/15 04:56:40 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 04:56:40 KserveLLMInferenceServiceDependencies: True - 2026/07/15 04:56:40 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/15 04:56:40 KserveReady: True - 2026/07/15 04:56:40 KueueReady: True - 2026/07/15 04:56:40 MLflowOperatorReady: True - 2026/07/15 04:56:40 ModelControllerReady: True - 2026/07/15 04:56:40 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/15 04:56:40 ModelsAsServiceReady: True - MaaS platform manifests applied and maas-api deployment is available 2026/07/15 04:56:40 OGXReady: False - Component ManagementState is set to Removed 2026/07/15 04:56:40 RayReady: False - Component ManagementState is set to Removed 2026/07/15 04:56:40 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 04:56:40 TrainerReady: False - Component ManagementState is set to Removed 2026/07/15 04:56:40 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 04:56:40 TrustyAIReady: True - 2026/07/15 04:56:40 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/15 04:56:40 === RECENT EVENTS === 2026/07/15 04:56:40 No recent events found in monitored namespaces 2026/07/15 04:56:40 === RESOURCE QUOTAS === 2026/07/15 04:56:40 No resource quotas found in monitored namespaces 2026/07/15 04:56:40 Classification for group 1: infrastructure/dsci-unhealthy (code=1009, confidence=medium) 2026/07/15 04:56:40 Evidence: DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"dsci-unhealthy","error_code":1009,"evidence":["DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False"],"confidence":"medium"} 2026/07/15 04:56:40 Diagnostics complete for failed test: group 1 controller_test.go:752: Stopping: group 1 test failed. 2026/07/15 04:56:40 Circuit breaker: failure classified as infrastructure/dsci-unhealthy (code=1009) 2026/07/15 04:56:40 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/15 05:00:42 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (1346.04s) === RUN TestOdhOperator 2026/07/15 04:45:06 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/15 05:07:43 TEST FAILURE DETECTED: components 2026/07/15 05:07:43 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/15 05:07:43 === CLUSTER STATE === 2026/07/15 05:07:43 Node: ip-10-0-1-23.us-east-2.compute.internal 2026/07/15 05:07:43 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 05:07:43 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 05:07:43 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 05:07:43 Condition Ready: True - kubelet is posting ready status 2026/07/15 05:07:43 Resources: 15500m CPU, 63182900Ki memory allocatable (of 16000m CPU, 64333876Ki memory capacity) 2026/07/15 05:07:43 Node: ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 05:07:43 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 05:07:43 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 05:07:43 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 05:07:43 Condition Ready: True - kubelet is posting ready status 2026/07/15 05:07:43 Resources: 15500m CPU, 63871012Ki memory allocatable (of 16000m CPU, 65021988Ki memory capacity) 2026/07/15 05:07:43 Node: ip-10-0-1-62.us-east-2.compute.internal 2026/07/15 05:07:43 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 05:07:43 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 05:07:43 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 05:07:43 Condition Ready: True - kubelet is posting ready status 2026/07/15 05:07:43 Resources: 15500m CPU, 63182896Ki memory allocatable (of 16000m CPU, 64333872Ki memory capacity) 2026/07/15 05:07:43 === DEPLOYMENTS === 2026/07/15 05:07:43 Namespace: kube-system 2026/07/15 05:07:43 Namespace: redhat-ods-applications 2026/07/15 05:07:43 Deployment dashboard-redirect: 2/2 (READY) 2026/07/15 05:07:43 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/15 05:07:43 Deployment kubeflow-trainer-controller-manager: 1/1 (READY) 2026/07/15 05:07:43 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/15 05:07:43 Deployment maas-postgres: 1/1 (READY) 2026/07/15 05:07:43 Deployment model-serving-api: 1/1 (READY) 2026/07/15 05:07:43 Deployment odh-model-controller: 1/1 (READY) 2026/07/15 05:07:43 === PODS === 2026/07/15 05:07:43 Namespace redhat-ods-applications: no problematic pods found 2026/07/15 05:07:43 Namespace kube-system: no problematic pods found 2026/07/15 05:07:43 === OPERATOR STATUS === 2026/07/15 05:07:43 Operator deployment: 3/3 ready 2026/07/15 05:07:43 === DSCI STATUS === 2026/07/15 05:07:43 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/15 05:07:43 === DSC STATUS === 2026/07/15 05:07:43 DSC default-dsc: 2026/07/15 05:07:43 Ready: True - 2026/07/15 05:07:43 ProvisioningSucceeded: True - 2026/07/15 05:07:43 ComponentsReady: True - 2026/07/15 05:07:43 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/15 05:07:43 DashboardReady: False - Component ManagementState is set to Removed 2026/07/15 05:07:43 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 05:07:43 KserveLLMInferenceServiceDependencies: True - 2026/07/15 05:07:43 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/15 05:07:43 KserveReady: True - 2026/07/15 05:07:43 KueueReady: False - Component ManagementState is set to Removed 2026/07/15 05:07:43 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 05:07:43 ModelControllerReady: True - 2026/07/15 05:07:43 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/15 05:07:43 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/15 05:07:43 OGXReady: False - Component ManagementState is set to Removed 2026/07/15 05:07:43 RayReady: False - Component ManagementState is set to Removed 2026/07/15 05:07:43 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 05:07:43 TrainerReady: True - 2026/07/15 05:07:43 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 05:07:43 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/15 05:07:43 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/15 05:07:43 === RECENT EVENTS === 2026/07/15 05:07:43 05:07:25 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-78dcbbdc6c-s559l_8d12dd0f-b97c-4626-af3b-d10ac5535957 became leader 2026/07/15 05:07:43 05:07:24 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-7747874bf-chntr_3a346f21-db2f-43af-a81a-13910f62830f became leader 2026/07/15 05:07:43 05:07:12 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-5b978c765f-r6qsv_b50c098b-7fc6-46a9-ad21-67d8d160cd3e became leader 2026/07/15 05:07:43 05:07:08 INFO Pod/kserve-controller-manager-7747874bf-chntr: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" in 556ms (556ms including waiting). Image size: 237680312 bytes. 2026/07/15 05:07:43 05:07:08 INFO Pod/kserve-controller-manager-7747874bf-chntr: Created - Created container: manager 2026/07/15 05:07:43 05:07:08 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-s559l: AddedInterface - Add eth0 [10.130.0.252/23] from ovn-kubernetes 2026/07/15 05:07:43 05:07:08 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-s559l: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" 2026/07/15 05:07:43 05:07:08 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-s559l: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" in 483ms (483ms including waiting). Image size: 235181729 bytes. 2026/07/15 05:07:43 05:07:08 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-s559l: Created - Created container: manager 2026/07/15 05:07:43 05:07:08 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-s559l: Started - Started container manager 2026/07/15 05:07:43 05:07:08 INFO Pod/kserve-controller-manager-7747874bf-chntr: Started - Started container manager 2026/07/15 05:07:43 05:07:07 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-s559l: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-78dcbbdc6c-s559l to ip-10-0-1-23.us-east-2.compute.internal 2026/07/15 05:07:43 05:07:07 INFO Pod/kserve-controller-manager-7747874bf-chntr: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-7747874bf-chntr to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 05:07:43 05:07:07 INFO Pod/kserve-controller-manager-7747874bf-chntr: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" 2026/07/15 05:07:43 05:07:07 INFO ReplicaSet/kserve-controller-manager-7747874bf: SuccessfulCreate - Created pod: kserve-controller-manager-7747874bf-chntr 2026/07/15 05:07:43 05:07:07 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-78dcbbdc6c from 0 to 1 2026/07/15 05:07:43 05:07:07 INFO Pod/kserve-controller-manager-7747874bf-chntr: AddedInterface - Add eth0 [10.129.1.47/23] from ovn-kubernetes 2026/07/15 05:07:43 05:07:07 INFO ReplicaSet/llmisvc-controller-manager-78dcbbdc6c: SuccessfulCreate - Created pod: llmisvc-controller-manager-78dcbbdc6c-s559l 2026/07/15 05:07:43 05:07:07 WARN Pod/llmisvc-controller-manager-78dcbbdc6c-s559l: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 2026/07/15 05:07:43 05:07:07 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7747874bf from 0 to 1 2026/07/15 05:07:43 05:06:56 INFO Pod/model-serving-api-67c44bccdf-pbvbp: Created - Created container: server 2026/07/15 05:07:43 05:06:56 INFO Pod/model-serving-api-67c44bccdf-pbvbp: AddedInterface - Add eth0 [10.130.0.251/23] from ovn-kubernetes 2026/07/15 05:07:43 05:06:56 INFO Pod/model-serving-api-67c44bccdf-pbvbp: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:226370d82da697958a0cd64b5cfa82f290ae1bb701e74b72739fcac0b38a834d" already present on machine 2026/07/15 05:07:43 05:06:56 INFO Pod/model-serving-api-67c44bccdf-pbvbp: Started - Started container server 2026/07/15 05:07:43 05:06:55 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 05:07:43 05:06:55 INFO Pod/model-serving-api-67c44bccdf-pbvbp: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-67c44bccdf-pbvbp to ip-10-0-1-23.us-east-2.compute.internal 2026/07/15 05:07:43 05:06:55 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-67c44bccdf from 0 to 1 2026/07/15 05:07:43 05:06:55 INFO Pod/odh-model-controller-5b978c765f-r6qsv: Started - Started container manager 2026/07/15 05:07:43 05:06:55 INFO ReplicaSet/model-serving-api-67c44bccdf: SuccessfulCreate - Created pod: model-serving-api-67c44bccdf-pbvbp 2026/07/15 05:07:43 05:06:55 WARN ReplicaSet/model-serving-api-67c44bccdf: FailedCreate - Error creating: pods "model-serving-api-67c44bccdf-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/15 05:07:43 05:06:55 INFO Pod/odh-model-controller-5b978c765f-r6qsv: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" in 563ms (563ms including waiting). Image size: 238719886 bytes. 2026/07/15 05:07:43 05:06:55 INFO Pod/odh-model-controller-5b978c765f-r6qsv: Created - Created container: manager 2026/07/15 05:07:43 05:06:54 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 05:07:43 05:06:54 INFO Pod/odh-model-controller-5b978c765f-r6qsv: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" 2026/07/15 05:07:43 05:06:54 WARN Pod/odh-model-controller-5b978c765f-r6qsv: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/15 05:07:43 05:06:54 INFO Pod/odh-model-controller-5b978c765f-r6qsv: AddedInterface - Add eth0 [10.129.1.46/23] from ovn-kubernetes 2026/07/15 05:07:43 05:06:53 INFO Pod/odh-model-controller-5b978c765f-r6qsv: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-5b978c765f-r6qsv to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 05:07:43 05:06:53 INFO ReplicaSet/odh-model-controller-5b978c765f: SuccessfulCreate - Created pod: odh-model-controller-5b978c765f-r6qsv 2026/07/15 05:07:43 05:06:53 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-5b978c765f from 0 to 1 2026/07/15 05:07:43 05:06:44 INFO Pod/model-serving-api-67c44bccdf-vw7nh: Killing - Stopping container server 2026/07/15 05:07:43 05:06:44 INFO Pod/odh-model-controller-5b978c765f-6rfh9: Killing - Stopping container manager 2026/07/15 05:07:43 05:06:44 INFO Pod/kserve-controller-manager-7747874bf-9xsp7: Killing - Stopping container manager 2026/07/15 05:07:43 05:06:44 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-frzsk: Killing - Stopping container manager 2026/07/15 05:07:43 05:06:26 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-78dcbbdc6c-frzsk_b568ba2b-1bc3-46d4-9db3-786acd02d736 became leader 2026/07/15 05:07:43 05:06:26 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-7747874bf-9xsp7_a47b5768-ad1a-463b-a703-f0c677c8f4a1 became leader 2026/07/15 05:07:43 05:06:11 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-5b978c765f-6rfh9_9b8d633a-5fe1-47b2-a589-52344ef20766 became leader 2026/07/15 05:07:43 05:06:09 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-frzsk: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" in 521ms (521ms including waiting). Image size: 235181729 bytes. 2026/07/15 05:07:43 05:06:09 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-frzsk: Created - Created container: manager 2026/07/15 05:07:43 05:06:09 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-frzsk: Started - Started container manager 2026/07/15 05:07:43 05:06:08 INFO Pod/kserve-controller-manager-7747874bf-9xsp7: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" in 545ms (545ms including waiting). Image size: 237680312 bytes. 2026/07/15 05:07:43 05:06:08 INFO Pod/kserve-controller-manager-7747874bf-9xsp7: Created - Created container: manager 2026/07/15 05:07:43 05:06:08 INFO Pod/kserve-controller-manager-7747874bf-9xsp7: Started - Started container manager 2026/07/15 05:07:43 05:06:08 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-frzsk: AddedInterface - Add eth0 [10.129.1.45/23] from ovn-kubernetes 2026/07/15 05:07:43 05:06:08 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-frzsk: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" 2026/07/15 05:07:43 05:06:07 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-frzsk: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-78dcbbdc6c-frzsk to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 05:07:43 05:06:07 INFO Pod/kserve-controller-manager-7747874bf-9xsp7: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-7747874bf-9xsp7 to ip-10-0-1-23.us-east-2.compute.internal 2026/07/15 05:07:43 05:06:07 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7747874bf from 0 to 1 2026/07/15 05:07:43 05:06:07 WARN Pod/llmisvc-controller-manager-78dcbbdc6c-frzsk: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 2026/07/15 05:07:43 05:06:07 INFO ReplicaSet/llmisvc-controller-manager-78dcbbdc6c: SuccessfulCreate - Created pod: llmisvc-controller-manager-78dcbbdc6c-frzsk 2026/07/15 05:07:43 05:06:07 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-78dcbbdc6c from 0 to 1 2026/07/15 05:07:43 05:06:07 INFO Pod/kserve-controller-manager-7747874bf-9xsp7: AddedInterface - Add eth0 [10.130.0.250/23] from ovn-kubernetes 2026/07/15 05:07:43 05:06:07 INFO Pod/kserve-controller-manager-7747874bf-9xsp7: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" 2026/07/15 05:07:43 05:06:07 INFO ReplicaSet/kserve-controller-manager-7747874bf: SuccessfulCreate - Created pod: kserve-controller-manager-7747874bf-9xsp7 2026/07/15 05:07:43 05:06:05 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-7dbc595499-tnd4w_27cfbdb1-6fe6-4a7b-a5a8-6fb1fca59d2f became leader 2026/07/15 05:07:43 05:05:55 INFO Pod/model-serving-api-67c44bccdf-vw7nh: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:226370d82da697958a0cd64b5cfa82f290ae1bb701e74b72739fcac0b38a834d" already present on machine 2026/07/15 05:07:43 05:05:55 INFO Pod/model-serving-api-67c44bccdf-vw7nh: AddedInterface - Add eth0 [10.130.0.249/23] from ovn-kubernetes 2026/07/15 05:07:43 05:05:55 INFO Pod/odh-model-controller-5b978c765f-6rfh9: Started - Started container manager 2026/07/15 05:07:43 05:05:55 INFO Pod/odh-model-controller-5b978c765f-6rfh9: Created - Created container: manager 2026/07/15 05:07:43 05:05:55 INFO Pod/model-serving-api-67c44bccdf-vw7nh: Started - Started container server 2026/07/15 05:07:43 05:05:55 INFO Pod/model-serving-api-67c44bccdf-vw7nh: Created - Created container: server 2026/07/15 05:07:43 05:05:54 WARN Pod/model-serving-api-67c44bccdf-vw7nh: FailedMount - MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 2026/07/15 05:07:43 05:05:54 INFO Pod/odh-model-controller-5b978c765f-6rfh9: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" in 618ms (618ms including waiting). Image size: 238719886 bytes. 2026/07/15 05:07:43 05:05:54 INFO Pod/odh-model-controller-5b978c765f-6rfh9: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" 2026/07/15 05:07:43 05:05:54 INFO Pod/odh-model-controller-5b978c765f-6rfh9: AddedInterface - Add eth0 [10.129.1.44/23] from ovn-kubernetes 2026/07/15 05:07:43 05:05:53 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 05:07:43 05:05:53 INFO Pod/model-serving-api-67c44bccdf-vw7nh: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-67c44bccdf-vw7nh to ip-10-0-1-23.us-east-2.compute.internal 2026/07/15 05:07:43 05:05:53 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 05:07:43 05:05:53 INFO Pod/odh-model-controller-5b978c765f-6rfh9: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-5b978c765f-6rfh9 to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 05:07:43 05:05:53 INFO ReplicaSet/odh-model-controller-5b978c765f: SuccessfulCreate - Created pod: odh-model-controller-5b978c765f-6rfh9 2026/07/15 05:07:43 05:05:53 WARN ReplicaSet/model-serving-api-67c44bccdf: FailedCreate - Error creating: pods "model-serving-api-67c44bccdf-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/15 05:07:43 05:05:53 WARN Pod/odh-model-controller-5b978c765f-6rfh9: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/15 05:07:43 05:05:53 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-5b978c765f from 0 to 1 2026/07/15 05:07:43 05:05:53 INFO ReplicaSet/model-serving-api-67c44bccdf: SuccessfulCreate - Created pod: model-serving-api-67c44bccdf-vw7nh 2026/07/15 05:07:43 05:05:53 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-67c44bccdf from 0 to 1 2026/07/15 05:07:43 05:05:51 INFO Pod/kserve-controller-manager-7747874bf-wkvbv: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" in 483ms (483ms including waiting). Image size: 237680312 bytes. 2026/07/15 05:07:43 05:05:51 INFO Pod/kserve-controller-manager-7747874bf-wkvbv: Started - Started container manager 2026/07/15 05:07:43 05:05:51 INFO Pod/kserve-controller-manager-7747874bf-wkvbv: Created - Created container: manager 2026/07/15 05:07:43 05:05:50 INFO Pod/kserve-controller-manager-7747874bf-wkvbv: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" 2026/07/15 05:07:43 05:05:50 INFO Pod/kserve-controller-manager-7747874bf-wkvbv: AddedInterface - Add eth0 [10.130.0.248/23] from ovn-kubernetes 2026/07/15 05:07:43 05:05:49 INFO Pod/kserve-controller-manager-7747874bf-wkvbv: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-7747874bf-wkvbv to ip-10-0-1-23.us-east-2.compute.internal 2026/07/15 05:07:43 05:05:49 INFO ReplicaSet/kserve-controller-manager-7747874bf: SuccessfulCreate - Created pod: kserve-controller-manager-7747874bf-wkvbv 2026/07/15 05:07:43 05:05:48 INFO Pod/kubeflow-trainer-controller-manager-7dbc595499-tnd4w: AddedInterface - Add eth0 [10.129.1.43/23] from ovn-kubernetes 2026/07/15 05:07:43 05:05:48 INFO Pod/kubeflow-trainer-controller-manager-7dbc595499-tnd4w: Pulled - Container image "quay.io/rhoai/odh-trainer-rhel9@sha256:595f456a6b3900dd112fa04416a2d91f448f8d68031b0c2de0d1dcb5fed28aa1" already present on machine 2026/07/15 05:07:43 05:05:48 INFO Pod/kubeflow-trainer-controller-manager-7dbc595499-tnd4w: Created - Created container: manager 2026/07/15 05:07:43 05:05:48 INFO Pod/kubeflow-trainer-controller-manager-7dbc595499-tnd4w: Started - Started container manager 2026/07/15 05:07:43 05:05:48 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7747874bf from 0 to 1 2026/07/15 05:07:43 05:05:47 INFO Pod/kubeflow-trainer-controller-manager-7dbc595499-tnd4w: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-7dbc595499-tnd4w to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 05:07:43 05:05:47 INFO ReplicaSet/kubeflow-trainer-controller-manager-7dbc595499: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-7dbc595499-tnd4w 2026/07/15 05:07:43 05:05:47 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-7dbc595499 from 0 to 1 2026/07/15 05:07:43 05:05:43 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-s8rcw: Killing - Stopping container manager 2026/07/15 05:07:43 05:05:43 INFO Pod/kserve-controller-manager-7747874bf-5wf9x: Killing - Stopping container manager 2026/07/15 05:07:43 05:05:43 INFO Pod/model-serving-api-67c44bccdf-m4pmg: Killing - Stopping container server 2026/07/15 05:07:43 05:05:43 INFO Pod/odh-model-controller-5b978c765f-ft6np: Killing - Stopping container manager 2026/07/15 05:07:43 05:05:32 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 05:07:43 05:05:32 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 05:07:43 05:05:32 INFO Pod/maas-controller-85c784d7b6-s6jdg: Killing - Stopping container manager 2026/07/15 05:07:43 05:05:32 INFO Pod/maas-api-5db58d77cb-sgtn4: Killing - Stopping container maas-api 2026/07/15 05:07:43 05:05:03 INFO Pod/maas-api-5db58d77cb-sgtn4: Started - Started container maas-api 2026/07/15 05:07:43 05:05:03 INFO Pod/maas-api-5db58d77cb-sgtn4: Created - Created container: maas-api 2026/07/15 05:07:43 05:05:03 INFO Pod/maas-api-5db58d77cb-sgtn4: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-api-rhel9@sha256:435d0a5a849ac8f7c05b7532b53158b29b2bead77295f2da203311777a0f0670" in 586ms (586ms including waiting). Image size: 174970750 bytes. 2026/07/15 05:07:43 05:05:02 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 05:07:43 05:05:02 INFO Pod/maas-api-5db58d77cb-sgtn4: Scheduled - Successfully assigned redhat-ods-applications/maas-api-5db58d77cb-sgtn4 to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 05:07:43 05:05:02 INFO Pod/maas-api-5db58d77cb-sgtn4: AddedInterface - Add eth0 [10.129.1.42/23] from ovn-kubernetes 2026/07/15 05:07:43 05:05:02 INFO Pod/maas-api-5db58d77cb-sgtn4: Pulling - Pulling image "quay.io/rhoai/odh-maas-api-rhel9@sha256:435d0a5a849ac8f7c05b7532b53158b29b2bead77295f2da203311777a0f0670" 2026/07/15 05:07:43 05:05:02 WARN ReplicaSet/maas-api-5db58d77cb: FailedCreate - Error creating: pods "maas-api-5db58d77cb-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 2026/07/15 05:07:43 05:05:02 INFO ReplicaSet/maas-api-5db58d77cb: SuccessfulCreate - Created pod: maas-api-5db58d77cb-sgtn4 2026/07/15 05:07:43 05:05:02 INFO Deployment/maas-api: ScalingReplicaSet - Scaled up replica set maas-api-5db58d77cb from 0 to 1 2026/07/15 05:07:43 05:04:44 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-7747874bf-5wf9x_b7feda40-4b7b-4d19-b0e5-14ac5958981c became leader 2026/07/15 05:07:43 05:04:43 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-78dcbbdc6c-s8rcw_6b1de5c8-b007-4cc8-8993-1fd991adb1b5 became leader 2026/07/15 05:07:43 05:04:31 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-5b978c765f-ft6np_5bff21d2-30fc-424c-8134-8b4d90f0c58d became leader 2026/07/15 05:07:43 05:04:31 INFO Lease/maas-controller.models-as-a-service.opendatahub.io: LeaderElection - maas-controller-85c784d7b6-s6jdg_b3c6cfd8-5d3b-4487-957b-56ca8c376ecb became leader 2026/07/15 05:07:43 05:04:27 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-s8rcw: Started - Started container manager 2026/07/15 05:07:43 05:04:27 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-s8rcw: Created - Created container: manager 2026/07/15 05:07:43 05:04:27 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-s8rcw: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" in 515ms (515ms including waiting). Image size: 235181729 bytes. 2026/07/15 05:07:43 05:04:26 INFO Pod/kserve-controller-manager-7747874bf-5wf9x: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" in 600ms (600ms including waiting). Image size: 237680312 bytes. 2026/07/15 05:07:43 05:04:26 INFO Pod/kserve-controller-manager-7747874bf-5wf9x: Created - Created container: manager 2026/07/15 05:07:43 05:04:26 INFO Pod/kserve-controller-manager-7747874bf-5wf9x: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" 2026/07/15 05:07:43 05:04:26 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-s8rcw: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" 2026/07/15 05:07:43 05:04:26 INFO Pod/kserve-controller-manager-7747874bf-5wf9x: AddedInterface - Add eth0 [10.129.1.41/23] from ovn-kubernetes 2026/07/15 05:07:43 05:04:26 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-s8rcw: AddedInterface - Add eth0 [10.130.0.246/23] from ovn-kubernetes 2026/07/15 05:07:43 05:04:26 INFO Pod/kserve-controller-manager-7747874bf-5wf9x: Started - Started container manager 2026/07/15 05:07:43 05:04:25 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-s8rcw: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-78dcbbdc6c-s8rcw to ip-10-0-1-23.us-east-2.compute.internal 2026/07/15 05:07:43 05:04:25 INFO Pod/kserve-controller-manager-7747874bf-5wf9x: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-7747874bf-5wf9x to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 05:07:43 05:04:25 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7747874bf from 0 to 1 2026/07/15 05:07:43 05:04:25 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-78dcbbdc6c from 0 to 1 2026/07/15 05:07:43 05:04:25 INFO ReplicaSet/kserve-controller-manager-7747874bf: SuccessfulCreate - Created pod: kserve-controller-manager-7747874bf-5wf9x 2026/07/15 05:07:43 05:04:25 INFO ReplicaSet/llmisvc-controller-manager-78dcbbdc6c: SuccessfulCreate - Created pod: llmisvc-controller-manager-78dcbbdc6c-s8rcw 2026/07/15 05:07:43 05:04:15 INFO Pod/model-serving-api-67c44bccdf-m4pmg: AddedInterface - Add eth0 [10.129.1.40/23] from ovn-kubernetes 2026/07/15 05:07:43 05:04:15 INFO Pod/model-serving-api-67c44bccdf-m4pmg: Started - Started container server 2026/07/15 05:07:43 05:04:15 INFO Pod/model-serving-api-67c44bccdf-m4pmg: Created - Created container: server 2026/07/15 05:07:43 05:04:15 INFO Pod/model-serving-api-67c44bccdf-m4pmg: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:226370d82da697958a0cd64b5cfa82f290ae1bb701e74b72739fcac0b38a834d" already present on machine 2026/07/15 05:07:43 05:04:14 INFO Pod/model-serving-api-67c44bccdf-m4pmg: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-67c44bccdf-m4pmg to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 05:07:43 05:04:14 INFO ReplicaSet/model-serving-api-67c44bccdf: SuccessfulCreate - Created pod: model-serving-api-67c44bccdf-m4pmg 2026/07/15 05:07:43 05:04:14 INFO Pod/odh-model-controller-5b978c765f-ft6np: Created - Created container: manager 2026/07/15 05:07:43 05:04:14 INFO Pod/odh-model-controller-5b978c765f-ft6np: Started - Started container manager 2026/07/15 05:07:43 05:04:14 INFO Pod/odh-model-controller-5b978c765f-ft6np: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" in 510ms (510ms including waiting). Image size: 238719886 bytes. 2026/07/15 05:07:43 05:04:14 WARN Pod/model-serving-api-67c44bccdf-m4pmg: FailedMount - MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 2026/07/15 05:07:43 05:04:14 WARN ReplicaSet/model-serving-api-67c44bccdf: FailedCreate - Error creating: pods "model-serving-api-67c44bccdf-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/15 05:07:43 05:04:14 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-67c44bccdf from 0 to 1 2026/07/15 05:07:43 05:04:13 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 05:07:43 05:04:13 INFO Pod/maas-controller-85c784d7b6-s6jdg: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:5effee51faa067115d737a4aa57473635576fb8232b4ea3cc39f76c0146abf65" 2026/07/15 05:07:43 05:04:13 INFO Pod/maas-controller-85c784d7b6-s6jdg: AddedInterface - Add eth0 [10.129.1.39/23] from ovn-kubernetes 2026/07/15 05:07:43 05:04:13 INFO Pod/odh-model-controller-5b978c765f-ft6np: AddedInterface - Add eth0 [10.129.1.38/23] from ovn-kubernetes 2026/07/15 05:07:43 05:04:13 INFO Pod/maas-controller-85c784d7b6-s6jdg: Started - Started container manager 2026/07/15 05:07:43 05:04:13 INFO Pod/maas-controller-85c784d7b6-s6jdg: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:5effee51faa067115d737a4aa57473635576fb8232b4ea3cc39f76c0146abf65" in 588ms (588ms including waiting). Image size: 212346876 bytes. 2026/07/15 05:07:43 05:04:13 WARN Pod/odh-model-controller-5b978c765f-ft6np: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/15 05:07:43 05:04:13 INFO Pod/maas-controller-85c784d7b6-s6jdg: Created - Created container: manager 2026/07/15 05:07:43 05:04:13 INFO Pod/odh-model-controller-5b978c765f-ft6np: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" 2026/07/15 05:07:43 05:04:12 INFO Pod/maas-controller-85c784d7b6-s6jdg: Scheduled - Successfully assigned redhat-ods-applications/maas-controller-85c784d7b6-s6jdg to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 05:07:43 05:04:12 INFO Pod/odh-model-controller-5b978c765f-ft6np: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-5b978c765f-ft6np to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 05:07:43 05:04:12 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-5b978c765f from 0 to 1 2026/07/15 05:07:43 05:04:12 INFO ReplicaSet/odh-model-controller-5b978c765f: SuccessfulCreate - Created pod: odh-model-controller-5b978c765f-ft6np 2026/07/15 05:07:43 05:04:12 INFO Deployment/maas-controller: ScalingReplicaSet - Scaled up replica set maas-controller-85c784d7b6 from 0 to 1 2026/07/15 05:07:43 05:04:12 INFO ReplicaSet/maas-controller-85c784d7b6: SuccessfulCreate - Created pod: maas-controller-85c784d7b6-s6jdg 2026/07/15 05:07:43 05:04:01 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-mkb44: Killing - Stopping container manager 2026/07/15 05:07:43 05:03:52 INFO Pod/kserve-controller-manager-7747874bf-nzgtq: Killing - Stopping container manager 2026/07/15 05:07:43 05:03:52 INFO Lease/b7e9931f.trustyai.opendatahub.io: LeaderElection - trustyai-service-operator-controller-manager-bc99cf85b-mkb44_d89ceca2-834c-484a-965a-5aaf7eca63cd became leader 2026/07/15 05:07:43 05:03:52 INFO Pod/odh-model-controller-5b978c765f-x4djx: Killing - Stopping container manager 2026/07/15 05:07:43 05:03:52 INFO Pod/model-serving-api-67c44bccdf-nnhq2: Killing - Stopping container server 2026/07/15 05:07:43 05:03:52 INFO Pod/maas-controller-85c784d7b6-cb4dl: Killing - Stopping container manager 2026/07/15 05:07:43 05:03:52 INFO Pod/maas-api-5db58d77cb-mh2lg: Killing - Stopping container maas-api 2026/07/15 05:07:43 05:03:51 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 05:07:43 05:03:51 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-qrndn: Killing - Stopping container manager 2026/07/15 05:07:43 05:03:51 WARN Pod/llmisvc-controller-manager-78dcbbdc6c-qrndn: Unhealthy - Readiness probe failed: Get "http://10.129.1.34:8081/readyz": dial tcp 10.129.1.34:8081: connect: connection refused 2026/07/15 05:07:43 05:03:36 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-mkb44: Created - Created container: manager 2026/07/15 05:07:43 05:03:36 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-mkb44: AddedInterface - Add eth0 [10.130.0.244/23] from ovn-kubernetes 2026/07/15 05:07:43 05:03:36 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-mkb44: Pulled - Container image "quay.io/rhoai/odh-trustyai-service-operator-rhel9@sha256:780ba93a8a4bf718fe19ad7bb9e4dec7b0ae79d5eb6d3e1e5060c3dc2260a515" already present on machine 2026/07/15 05:07:43 05:03:36 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-mkb44: Started - Started container manager 2026/07/15 05:07:43 05:03:35 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-mkb44: Scheduled - Successfully assigned redhat-ods-applications/trustyai-service-operator-controller-manager-bc99cf85b-mkb44 to ip-10-0-1-23.us-east-2.compute.internal 2026/07/15 05:07:43 05:03:35 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-5r7mm: Killing - Stopping container manager 2026/07/15 05:07:43 05:03:35 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-kb8kt: Killing - Stopping container manager 2026/07/15 05:07:43 05:03:35 INFO ReplicaSet/trustyai-service-operator-controller-manager-bc99cf85b: SuccessfulCreate - Created pod: trustyai-service-operator-controller-manager-bc99cf85b-mkb44 2026/07/15 05:07:43 05:03:35 INFO Deployment/trustyai-service-operator-controller-manager: ScalingReplicaSet - Scaled up replica set trustyai-service-operator-controller-manager-bc99cf85b from 0 to 1 2026/07/15 05:07:43 05:03:05 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 05:07:43 05:03:00 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 05:07:43 05:02:55 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 05:07:43 === RESOURCE QUOTAS === 2026/07/15 05:07:43 No resource quotas found in monitored namespaces 2026/07/15 05:07:43 Classification for components: infrastructure/storage (code=1006, confidence=medium) 2026/07/15 05:07:43 Evidence: event Pod/llmisvc-controller-manager-78dcbbdc6c-s559l: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/llmisvc-controller-manager-78dcbbdc6c-s559l: FailedMount - MountVolume.SetUp failed for volume \"cert\" : secret \"llmisvc-webhook-server-cert\" not found"],"confidence":"medium"} 2026/07/15 05:07:43 Diagnostics complete for failed test: components controller_test.go:483: Stopping: components test failed. 2026/07/15 05:07:43 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/15 05:07:43 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/15 05:07:44 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (1358.31s)