=== RUN TestOdhOperator/components/group_1/kserve/Validate_component_spec testing.go:1811: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_1/kserve/Validate_component_spec (600.03s) === RUN TestOdhOperator/components/group_1/kserve === PAUSE TestOdhOperator/components/group_1/kserve === CONT TestOdhOperator/components/group_1/kserve 2026-07-14T23:52:22Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.1"}, "Cluster": {"type":"OpenShift","version":"4.21.19"}} test_context_test.go:223: Timed out after 600.007s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:114 with: Expected resource 'default-kserve' of kind 'Kserve' to match condition {.spec.nim.managementState == "" []}. Expected <string>: map[apiVersion:components.platform.opendatahub.io/v1alpha1 kind:Kserve metadata:map[annotations:map[component.opendatahub.io/management-state:Managed platform.opendatahub.io/instance.generation:37 platform.opendatahub.io/instance.name:default-dsc platform.opendatahub.io/instance.uid:9013f03c-d6c8-43c1-bdd4-c3e666f6094f platform.opendatahub.io/type:OpenShift AI Self-Managed platform.opendatahub.io/version:3.5.0-ea.1] creationTimestamp:2026-07-14T23:52:34Z generation:1 labels:map[platform.opendatahub.io/part-of:datasciencecluster] managedFields:[map[apiVersion:components.platform.opendatahub.io/v1alpha1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:component.opendatahub.io/management-state:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:labels:map[f:platform.opendatahub.io/part-of:map[]] f:ownerReferences:map[k:{"uid":"9013f03c-d6c8-43c1-bdd4-c3e666f6094f"}:map[]]] f:spec:map[f:modelsAsService:map[] f:nim:map[] f:rawDeploymentServiceConfig:map[] f:wva:map[]]] manager:datasciencecluster operation:Apply time:2026-07-14T23:54:53Z] map[apiVersion:components.platform.opendatahub.io/v1alpha1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:component.opendatahub.io/management-state:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:labels:map[f:platform.opendatahub.io/part-of:map[]] f:ownerReferences:map[k:{"uid":"9013f03c-d6c8-43c1-bdd4-c3e666f6094f"}:map[]]] f:status:map[f:conditions:map[] f:observedGeneration:map[] f:phase:map[] f:releases:map[k:{"name":"KServe"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-inference-scheduler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-workload-variant-autoscaler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"vLLM"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]]] manager:kserve operation:Apply subresource:status time:2026-07-15T00:03:34Z]] name:default-kserve ownerReferences:[map[apiVersion:datasciencecluster.opendatahub.io/v2 blockOwnerDeletion:true controller:true kind:DataScienceCluster name:default-dsc uid:9013f03c-d6c8-43c1-bdd4-c3e666f6094f]] resourceVersion:32814346 uid:d90ed12e-b71a-4e8e-b5c3-d10cb4984caa] spec:map[modelsAsService:map[managementState:Removed] nim:map[airGapped:false managementState:Managed] rawDeploymentServiceConfig:Headless wva:map[managementState:Removed]] status:map[conditions:[map[lastTransitionTime:2026-07-15T00:03:33Z status:True type:Ready] map[lastTransitionTime:2026-07-15T00:03:34Z observedGeneration:1 status:True type:ProvisioningSucceeded] map[lastTransitionTime:2026-07-15T00:03:34Z observedGeneration:1 status:True type:DeploymentsAvailable] map[lastTransitionTime:2026-07-15T00:03:33Z status:True type:DependenciesAvailable] map[lastTransitionTime:2026-07-15T00:03:33Z status:True type:KserveLLMInferenceServiceDependencies] map[lastTransitionTime:2026-07-15T00:03:33Z status:True type:KserveLLMInferenceServiceWideEPDependencies]] observedGeneration:1 phase:Ready releases:[map[name:KServe repoUrl:https://github.com/kserve/kserve/ version:v0.17.0] map[name:vLLM repoUrl:https://github.com/vllm-project/vllm version:v0.18.0] map[name:llm-d-inference-scheduler repoUrl:https://github.com/llm-d/llm-d-inference-scheduler version:v0.7.1] map[name:llm-d-workload-variant-autoscaler repoUrl:https://github.com/llm-d/llm-d-workload-variant-autoscaler version:v0.6.0]]]] to match expression <string>: .spec.nim.managementState == "" --- FAIL: TestOdhOperator/components/group_1/kserve (681.27s) === RUN TestOdhOperator/components/group_1 --- FAIL: TestOdhOperator/components/group_1 (0.00s) === RUN TestOdhOperator/components 2026/07/15 00:03:43 TEST FAILURE DETECTED: group 1 2026/07/15 00:03:43 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 1) === 2026/07/15 00:03:43 === CLUSTER STATE === 2026/07/15 00:03:43 Node: ip-10-0-1-23.us-east-2.compute.internal 2026/07/15 00:03:43 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 00:03:43 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 00:03:43 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 00:03:43 Condition Ready: True - kubelet is posting ready status 2026/07/15 00:03:43 Resources: 15500m CPU, 63182900Ki memory allocatable (of 16000m CPU, 64333876Ki memory capacity) 2026/07/15 00:03:43 Node: ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 00:03:43 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 00:03:43 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 00:03:43 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 00:03:43 Condition Ready: True - kubelet is posting ready status 2026/07/15 00:03:43 Resources: 15500m CPU, 63871012Ki memory allocatable (of 16000m CPU, 65021988Ki memory capacity) 2026/07/15 00:03:43 Node: ip-10-0-1-62.us-east-2.compute.internal 2026/07/15 00:03:43 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 00:03:43 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 00:03:43 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 00:03:43 Condition Ready: True - kubelet is posting ready status 2026/07/15 00:03:43 Resources: 15500m CPU, 63182896Ki memory allocatable (of 16000m CPU, 64333872Ki memory capacity) 2026/07/15 00:03:43 === DEPLOYMENTS === 2026/07/15 00:03:43 Namespace: redhat-ods-applications 2026/07/15 00:03:43 Deployment dashboard-redirect: 2/2 (READY) 2026/07/15 00:03:43 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/15 00:03:43 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/15 00:03:43 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/15 00:03:43 Deployment model-serving-api: 1/1 (READY) 2026/07/15 00:03:43 Deployment odh-model-controller: 1/1 (READY) 2026/07/15 00:03:43 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/15 00:03:43 Namespace: kube-system 2026/07/15 00:03:43 === PODS === 2026/07/15 00:03:43 Failed to collect pod data: redhat-ods-applications/trustyai-service-operator-controller-manager-bc99cf85b-749kd: container manager restarts=1 2026/07/15 00:03:43 === OPERATOR STATUS === 2026/07/15 00:03:43 Operator deployment: 3/3 ready 2026/07/15 00:03:43 === DSCI STATUS === 2026/07/15 00:03:43 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/15 00:03:43 === DSC STATUS === 2026/07/15 00:03:43 DSC default-dsc: 2026/07/15 00:03:43 Ready: True - 2026/07/15 00:03:43 ProvisioningSucceeded: True - 2026/07/15 00:03:43 ComponentsReady: True - 2026/07/15 00:03:43 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/15 00:03:43 DashboardReady: False - Component ManagementState is set to Removed 2026/07/15 00:03:43 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 00:03:43 KserveLLMInferenceServiceDependencies: True - 2026/07/15 00:03:43 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/15 00:03:43 KserveReady: True - 2026/07/15 00:03:43 KueueReady: True - 2026/07/15 00:03:43 MLflowOperatorReady: True - 2026/07/15 00:03:43 ModelControllerReady: True - 2026/07/15 00:03:43 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/15 00:03:43 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/15 00:03:43 OGXReady: False - Component ManagementState is set to Removed 2026/07/15 00:03:43 RayReady: False - Component ManagementState is set to Removed 2026/07/15 00:03:43 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 00:03:43 TrainerReady: False - Component ManagementState is set to Removed 2026/07/15 00:03:43 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 00:03:43 TrustyAIReady: True - 2026/07/15 00:03:43 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/15 00:03:43 === RECENT EVENTS === 2026/07/15 00:03:43 00:01:48 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 00:03:43 00:01:48 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 00:03:43 00:01:47 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 00:03:43 00:01:47 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 00:03:43 === RESOURCE QUOTAS === 2026/07/15 00:03:43 No resource quotas found in monitored namespaces 2026/07/15 00:03:43 Classification for group 1: infrastructure/dsci-unhealthy (code=1009, confidence=medium) 2026/07/15 00:03:43 Evidence: DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"dsci-unhealthy","error_code":1009,"evidence":["DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False"],"confidence":"medium"} 2026/07/15 00:03:43 Diagnostics complete for failed test: group 1 controller_test.go:752: Stopping: group 1 test failed. 2026/07/15 00:03:43 Circuit breaker: failure classified as infrastructure/dsci-unhealthy (code=1009) 2026/07/15 00:03:43 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/15 00:07:15 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (1457.31s) === RUN TestOdhOperator 2026/07/14 23:50:38 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/15 00:15:06 TEST FAILURE DETECTED: components 2026/07/15 00:15:06 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/15 00:15:07 === CLUSTER STATE === 2026/07/15 00:15:07 Node: ip-10-0-1-23.us-east-2.compute.internal 2026/07/15 00:15:07 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 00:15:07 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 00:15:07 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 00:15:07 Condition Ready: True - kubelet is posting ready status 2026/07/15 00:15:07 Resources: 15500m CPU, 63182900Ki memory allocatable (of 16000m CPU, 64333876Ki memory capacity) 2026/07/15 00:15:07 Node: ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 00:15:07 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 00:15:07 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 00:15:07 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 00:15:07 Condition Ready: True - kubelet is posting ready status 2026/07/15 00:15:07 Resources: 15500m CPU, 63871012Ki memory allocatable (of 16000m CPU, 65021988Ki memory capacity) 2026/07/15 00:15:07 Node: ip-10-0-1-62.us-east-2.compute.internal 2026/07/15 00:15:07 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 00:15:07 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 00:15:07 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 00:15:07 Condition Ready: True - kubelet is posting ready status 2026/07/15 00:15:07 Resources: 15500m CPU, 63182896Ki memory allocatable (of 16000m CPU, 64333872Ki memory capacity) 2026/07/15 00:15:07 === DEPLOYMENTS === 2026/07/15 00:15:07 Namespace: redhat-ods-applications 2026/07/15 00:15:07 Deployment dashboard-redirect: 2/2 (READY) 2026/07/15 00:15:07 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/15 00:15:07 Deployment kubeflow-trainer-controller-manager: 1/1 (READY) 2026/07/15 00:15:07 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/15 00:15:07 Deployment maas-postgres: 1/1 (READY) 2026/07/15 00:15:07 Deployment model-serving-api: 1/1 (READY) 2026/07/15 00:15:07 Deployment odh-model-controller: 1/1 (READY) 2026/07/15 00:15:07 Namespace: kube-system 2026/07/15 00:15:07 === PODS === 2026/07/15 00:15:07 Namespace redhat-ods-applications: no problematic pods found 2026/07/15 00:15:07 Namespace kube-system: no problematic pods found 2026/07/15 00:15:07 === OPERATOR STATUS === 2026/07/15 00:15:07 Operator deployment: 3/3 ready 2026/07/15 00:15:07 === DSCI STATUS === 2026/07/15 00:15:07 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/15 00:15:07 === DSC STATUS === 2026/07/15 00:15:07 DSC default-dsc: 2026/07/15 00:15:07 Ready: True - 2026/07/15 00:15:07 ProvisioningSucceeded: True - 2026/07/15 00:15:07 ComponentsReady: True - 2026/07/15 00:15:07 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/15 00:15:07 DashboardReady: False - Component ManagementState is set to Removed 2026/07/15 00:15:07 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 00:15:07 KserveLLMInferenceServiceDependencies: True - 2026/07/15 00:15:07 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/15 00:15:07 KserveReady: True - 2026/07/15 00:15:07 KueueReady: False - Component ManagementState is set to Removed 2026/07/15 00:15:07 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 00:15:07 ModelControllerReady: True - 2026/07/15 00:15:07 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/15 00:15:07 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/15 00:15:07 OGXReady: False - Component ManagementState is set to Removed 2026/07/15 00:15:07 RayReady: False - Component ManagementState is set to Removed 2026/07/15 00:15:07 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 00:15:07 TrainerReady: True - 2026/07/15 00:15:07 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 00:15:07 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/15 00:15:07 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/15 00:15:07 === RECENT EVENTS === 2026/07/15 00:15:07 00:15:05 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 00:15:07 00:15:00 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 00:15:07 00:14:55 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 00:15:07 00:14:49 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-7747874bf-tqwfg_c665a439-1ca4-4c2f-ba5e-a5f189af2308 became leader 2026/07/15 00:15:07 00:14:47 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-78dcbbdc6c-8q69m_fc4c4293-0d31-4e82-97a6-fa312933e72a became leader 2026/07/15 00:15:07 00:14:36 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-5b978c765f-gknz9_00fea609-c659-45ea-98c0-dbde679806f0 became leader 2026/07/15 00:15:07 00:14:31 INFO Pod/kserve-controller-manager-7747874bf-tqwfg: Started - Started container manager 2026/07/15 00:15:07 00:14:31 INFO Pod/kserve-controller-manager-7747874bf-tqwfg: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" in 570ms (570ms including waiting). Image size: 237680312 bytes. 2026/07/15 00:15:07 00:14:31 INFO Pod/kserve-controller-manager-7747874bf-tqwfg: Created - Created container: manager 2026/07/15 00:15:07 00:14:30 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-8q69m: AddedInterface - Add eth0 [10.129.0.252/23] from ovn-kubernetes 2026/07/15 00:15:07 00:14:30 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-8q69m: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" in 696ms (696ms including waiting). Image size: 235181729 bytes. 2026/07/15 00:15:07 00:14:30 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-8q69m: Created - Created container: manager 2026/07/15 00:15:07 00:14:30 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-8q69m: Started - Started container manager 2026/07/15 00:15:07 00:14:30 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-8q69m: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" 2026/07/15 00:15:07 00:14:30 INFO Pod/kserve-controller-manager-7747874bf-tqwfg: AddedInterface - Add eth0 [10.130.0.161/23] from ovn-kubernetes 2026/07/15 00:15:07 00:14:30 INFO Pod/kserve-controller-manager-7747874bf-tqwfg: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" 2026/07/15 00:15:07 00:14:29 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-8q69m: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-78dcbbdc6c-8q69m to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 00:15:07 00:14:29 INFO Pod/kserve-controller-manager-7747874bf-tqwfg: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-7747874bf-tqwfg to ip-10-0-1-23.us-east-2.compute.internal 2026/07/15 00:15:07 00:14:29 INFO ReplicaSet/llmisvc-controller-manager-78dcbbdc6c: SuccessfulCreate - Created pod: llmisvc-controller-manager-78dcbbdc6c-8q69m 2026/07/15 00:15:07 00:14:29 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7747874bf from 0 to 1 2026/07/15 00:15:07 00:14:29 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-78dcbbdc6c from 0 to 1 2026/07/15 00:15:07 00:14:29 WARN Pod/kserve-controller-manager-7747874bf-tqwfg: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 2026/07/15 00:15:07 00:14:29 INFO ReplicaSet/kserve-controller-manager-7747874bf: SuccessfulCreate - Created pod: kserve-controller-manager-7747874bf-tqwfg 2026/07/15 00:15:07 00:14:19 INFO Pod/model-serving-api-67c44bccdf-mgfcz: Created - Created container: server 2026/07/15 00:15:07 00:14:19 INFO Pod/model-serving-api-67c44bccdf-mgfcz: AddedInterface - Add eth0 [10.129.0.251/23] from ovn-kubernetes 2026/07/15 00:15:07 00:14:19 INFO Pod/model-serving-api-67c44bccdf-mgfcz: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:226370d82da697958a0cd64b5cfa82f290ae1bb701e74b72739fcac0b38a834d" already present on machine 2026/07/15 00:15:07 00:14:19 WARN Pod/model-serving-api-67c44bccdf-mgfcz: FailedMount - MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 2026/07/15 00:15:07 00:14:19 INFO Pod/model-serving-api-67c44bccdf-mgfcz: Started - Started container server 2026/07/15 00:15:07 00:14:19 INFO Pod/odh-model-controller-5b978c765f-gknz9: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" in 530ms (530ms including waiting). Image size: 238719886 bytes. 2026/07/15 00:15:07 00:14:19 INFO Pod/odh-model-controller-5b978c765f-gknz9: Created - Created container: manager 2026/07/15 00:15:07 00:14:19 INFO Pod/odh-model-controller-5b978c765f-gknz9: Started - Started container manager 2026/07/15 00:15:07 00:14:18 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 00:15:07 00:14:18 INFO Pod/model-serving-api-67c44bccdf-mgfcz: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-67c44bccdf-mgfcz to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 00:15:07 00:14:18 INFO Pod/odh-model-controller-5b978c765f-gknz9: AddedInterface - Add eth0 [10.129.0.250/23] from ovn-kubernetes 2026/07/15 00:15:07 00:14:18 INFO Pod/odh-model-controller-5b978c765f-gknz9: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" 2026/07/15 00:15:07 00:14:18 WARN ReplicaSet/model-serving-api-67c44bccdf: FailedCreate - Error creating: pods "model-serving-api-67c44bccdf-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/15 00:15:07 00:14:18 INFO ReplicaSet/model-serving-api-67c44bccdf: SuccessfulCreate - Created pod: model-serving-api-67c44bccdf-mgfcz 2026/07/15 00:15:07 00:14:18 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-67c44bccdf from 0 to 1 2026/07/15 00:15:07 00:14:17 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 00:15:07 00:14:17 INFO Pod/odh-model-controller-5b978c765f-gknz9: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-5b978c765f-gknz9 to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 00:15:07 00:14:17 WARN Pod/odh-model-controller-5b978c765f-gknz9: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/15 00:15:07 00:14:17 INFO ReplicaSet/odh-model-controller-5b978c765f: SuccessfulCreate - Created pod: odh-model-controller-5b978c765f-gknz9 2026/07/15 00:15:07 00:14:17 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-5b978c765f from 0 to 1 2026/07/15 00:15:07 00:14:07 INFO Pod/model-serving-api-67c44bccdf-gbzmt: Killing - Stopping container server 2026/07/15 00:15:07 00:14:07 INFO Pod/odh-model-controller-5b978c765f-6mhlh: Killing - Stopping container manager 2026/07/15 00:15:07 00:14:07 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-w4pbc: Killing - Stopping container manager 2026/07/15 00:15:07 00:14:07 INFO Pod/kserve-controller-manager-7747874bf-ngtkj: Killing - Stopping container manager 2026/07/15 00:15:07 00:14:06 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 00:15:07 00:13:48 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-78dcbbdc6c-w4pbc_2bd000c6-87b8-4600-a12a-940dbee64632 became leader 2026/07/15 00:15:07 00:13:47 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-7747874bf-ngtkj_4002c15c-1335-40f9-916b-3927a0eab140 became leader 2026/07/15 00:15:07 00:13:34 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-5b978c765f-6mhlh_d90e8f81-9994-435d-981e-999f2329e9b9 became leader 2026/07/15 00:15:07 00:13:30 INFO Pod/kserve-controller-manager-7747874bf-ngtkj: Created - Created container: manager 2026/07/15 00:15:07 00:13:30 INFO Pod/kserve-controller-manager-7747874bf-ngtkj: Started - Started container manager 2026/07/15 00:15:07 00:13:29 INFO Pod/kserve-controller-manager-7747874bf-ngtkj: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" in 540ms (540ms including waiting). Image size: 237680312 bytes. 2026/07/15 00:15:07 00:13:29 INFO Pod/kserve-controller-manager-7747874bf-ngtkj: AddedInterface - Add eth0 [10.130.0.160/23] from ovn-kubernetes 2026/07/15 00:15:07 00:13:29 INFO Pod/kserve-controller-manager-7747874bf-ngtkj: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" 2026/07/15 00:15:07 00:13:29 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-w4pbc: Started - Started container manager 2026/07/15 00:15:07 00:13:29 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-w4pbc: AddedInterface - Add eth0 [10.129.0.249/23] from ovn-kubernetes 2026/07/15 00:15:07 00:13:29 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-w4pbc: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" 2026/07/15 00:15:07 00:13:29 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-w4pbc: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" in 689ms (689ms including waiting). Image size: 235181729 bytes. 2026/07/15 00:15:07 00:13:29 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-w4pbc: Created - Created container: manager 2026/07/15 00:15:07 00:13:28 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-w4pbc: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-78dcbbdc6c-w4pbc to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 00:15:07 00:13:28 INFO Pod/kserve-controller-manager-7747874bf-ngtkj: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-7747874bf-ngtkj to ip-10-0-1-23.us-east-2.compute.internal 2026/07/15 00:15:07 00:13:28 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-78dcbbdc6c from 0 to 1 2026/07/15 00:15:07 00:13:28 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-7dbc595499-wdcgg_23f4125d-e83d-46cd-9ba6-c15f5a2e8a8a became leader 2026/07/15 00:15:07 00:13:28 INFO ReplicaSet/kserve-controller-manager-7747874bf: SuccessfulCreate - Created pod: kserve-controller-manager-7747874bf-ngtkj 2026/07/15 00:15:07 00:13:28 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7747874bf from 0 to 1 2026/07/15 00:15:07 00:13:28 INFO ReplicaSet/llmisvc-controller-manager-78dcbbdc6c: SuccessfulCreate - Created pod: llmisvc-controller-manager-78dcbbdc6c-w4pbc 2026/07/15 00:15:07 00:13:19 INFO Pod/model-serving-api-67c44bccdf-gbzmt: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:226370d82da697958a0cd64b5cfa82f290ae1bb701e74b72739fcac0b38a834d" already present on machine 2026/07/15 00:15:07 00:13:19 INFO Pod/model-serving-api-67c44bccdf-gbzmt: Created - Created container: server 2026/07/15 00:15:07 00:13:19 INFO Pod/model-serving-api-67c44bccdf-gbzmt: Started - Started container server 2026/07/15 00:15:07 00:13:19 INFO Pod/model-serving-api-67c44bccdf-gbzmt: AddedInterface - Add eth0 [10.129.0.248/23] from ovn-kubernetes 2026/07/15 00:15:07 00:13:17 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 00:15:07 00:13:17 INFO Pod/model-serving-api-67c44bccdf-gbzmt: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-67c44bccdf-gbzmt to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 00:15:07 00:13:17 INFO Pod/odh-model-controller-5b978c765f-6mhlh: Created - Created container: manager 2026/07/15 00:15:07 00:13:17 WARN Pod/model-serving-api-67c44bccdf-gbzmt: FailedMount - MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 2026/07/15 00:15:07 00:13:17 INFO Pod/odh-model-controller-5b978c765f-6mhlh: Started - Started container manager 2026/07/15 00:15:07 00:13:17 INFO Pod/odh-model-controller-5b978c765f-6mhlh: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" in 1.064s (1.064s including waiting). Image size: 238719886 bytes. 2026/07/15 00:15:07 00:13:17 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-67c44bccdf from 0 to 1 2026/07/15 00:15:07 00:13:17 INFO ReplicaSet/model-serving-api-67c44bccdf: SuccessfulCreate - Created pod: model-serving-api-67c44bccdf-gbzmt 2026/07/15 00:15:07 00:13:17 WARN ReplicaSet/model-serving-api-67c44bccdf: FailedCreate - Error creating: pods "model-serving-api-67c44bccdf-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/15 00:15:07 00:13:16 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 00:15:07 00:13:16 INFO Pod/odh-model-controller-5b978c765f-6mhlh: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-5b978c765f-6mhlh to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 00:15:07 00:13:16 INFO Pod/odh-model-controller-5b978c765f-6mhlh: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" 2026/07/15 00:15:07 00:13:16 INFO Pod/odh-model-controller-5b978c765f-6mhlh: AddedInterface - Add eth0 [10.129.0.247/23] from ovn-kubernetes 2026/07/15 00:15:07 00:13:16 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-5b978c765f from 0 to 1 2026/07/15 00:15:07 00:13:16 INFO ReplicaSet/odh-model-controller-5b978c765f: SuccessfulCreate - Created pod: odh-model-controller-5b978c765f-6mhlh 2026/07/15 00:15:07 00:13:11 INFO Pod/kubeflow-trainer-controller-manager-7dbc595499-wdcgg: Created - Created container: manager 2026/07/15 00:15:07 00:13:11 INFO Pod/kubeflow-trainer-controller-manager-7dbc595499-wdcgg: AddedInterface - Add eth0 [10.129.0.246/23] from ovn-kubernetes 2026/07/15 00:15:07 00:13:11 INFO Pod/kubeflow-trainer-controller-manager-7dbc595499-wdcgg: Pulled - Container image "quay.io/rhoai/odh-trainer-rhel9@sha256:595f456a6b3900dd112fa04416a2d91f448f8d68031b0c2de0d1dcb5fed28aa1" already present on machine 2026/07/15 00:15:07 00:13:11 INFO Pod/kubeflow-trainer-controller-manager-7dbc595499-wdcgg: Started - Started container manager 2026/07/15 00:15:07 00:13:10 INFO Pod/kubeflow-trainer-controller-manager-7dbc595499-wdcgg: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-7dbc595499-wdcgg to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 00:15:07 00:13:10 WARN ReplicaSet/kserve-controller-manager-7747874bf: FailedCreate - Error creating: pods "kserve-controller-manager-7747874bf-" is forbidden: error looking up service account redhat-ods-applications/kserve-controller-manager: serviceaccount "kserve-controller-manager" not found 2026/07/15 00:15:07 00:13:10 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-7dbc595499 from 0 to 1 2026/07/15 00:15:07 00:13:10 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7747874bf from 0 to 1 2026/07/15 00:15:07 00:13:10 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-78dcbbdc6c from 0 to 1 2026/07/15 00:15:07 00:13:10 INFO ReplicaSet/kubeflow-trainer-controller-manager-7dbc595499: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-7dbc595499-wdcgg 2026/07/15 00:15:07 00:13:10 WARN ReplicaSet/llmisvc-controller-manager-78dcbbdc6c: FailedCreate - Error creating: pods "llmisvc-controller-manager-78dcbbdc6c-" is forbidden: error looking up service account redhat-ods-applications/llmisvc-controller-manager: serviceaccount "llmisvc-controller-manager" not found 2026/07/15 00:15:07 00:13:06 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-b5tgs: Killing - Stopping container manager 2026/07/15 00:15:07 00:13:06 INFO Pod/model-serving-api-67c44bccdf-tzjzk: Killing - Stopping container server 2026/07/15 00:15:07 00:13:06 INFO Pod/odh-model-controller-5b978c765f-npm2r: Killing - Stopping container manager 2026/07/15 00:15:07 00:13:06 INFO Pod/kserve-controller-manager-7747874bf-gwjgq: Killing - Stopping container manager 2026/07/15 00:15:07 00:12:55 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 00:15:07 00:12:55 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 00:15:07 00:12:55 INFO Pod/maas-controller-85c784d7b6-cnbhs: Killing - Stopping container manager 2026/07/15 00:15:07 00:12:55 INFO Pod/maas-api-5db58d77cb-4ccd8: Killing - Stopping container maas-api 2026/07/15 00:15:07 00:12:30 INFO Pod/maas-api-5db58d77cb-4ccd8: Created - Created container: maas-api 2026/07/15 00:15:07 00:12:30 INFO Pod/maas-api-5db58d77cb-4ccd8: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-api-rhel9@sha256:435d0a5a849ac8f7c05b7532b53158b29b2bead77295f2da203311777a0f0670" in 1.853s (1.853s including waiting). Image size: 174970750 bytes. 2026/07/15 00:15:07 00:12:30 INFO Pod/maas-api-5db58d77cb-4ccd8: Started - Started container maas-api 2026/07/15 00:15:07 00:12:28 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 00:15:07 00:12:28 INFO Pod/maas-api-5db58d77cb-4ccd8: AddedInterface - Add eth0 [10.129.0.245/23] from ovn-kubernetes 2026/07/15 00:15:07 00:12:28 INFO Pod/maas-api-5db58d77cb-4ccd8: Pulling - Pulling image "quay.io/rhoai/odh-maas-api-rhel9@sha256:435d0a5a849ac8f7c05b7532b53158b29b2bead77295f2da203311777a0f0670" 2026/07/15 00:15:07 00:12:27 INFO Pod/maas-api-5db58d77cb-4ccd8: Scheduled - Successfully assigned redhat-ods-applications/maas-api-5db58d77cb-4ccd8 to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 00:15:07 00:12:27 INFO ReplicaSet/maas-api-5db58d77cb: SuccessfulCreate - Created pod: maas-api-5db58d77cb-4ccd8 2026/07/15 00:15:07 00:12:27 WARN ReplicaSet/maas-api-5db58d77cb: FailedCreate - Error creating: pods "maas-api-5db58d77cb-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 2026/07/15 00:15:07 00:12:27 INFO Deployment/maas-api: ScalingReplicaSet - Scaled up replica set maas-api-5db58d77cb from 0 to 1 2026/07/15 00:15:07 00:11:57 INFO Lease/maas-controller.models-as-a-service.opendatahub.io: LeaderElection - maas-controller-85c784d7b6-cnbhs_166eaf2f-bb89-47c7-9d82-d461bbffaf31 became leader 2026/07/15 00:15:07 00:11:56 INFO Pod/maas-controller-85c784d7b6-cnbhs: Started - Started container manager 2026/07/15 00:15:07 00:11:56 INFO Pod/maas-controller-85c784d7b6-cnbhs: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:5effee51faa067115d737a4aa57473635576fb8232b4ea3cc39f76c0146abf65" in 772ms (772ms including waiting). Image size: 212346876 bytes. 2026/07/15 00:15:07 00:11:56 INFO Pod/maas-controller-85c784d7b6-cnbhs: Created - Created container: manager 2026/07/15 00:15:07 00:11:55 INFO Pod/maas-controller-85c784d7b6-cnbhs: Scheduled - Successfully assigned redhat-ods-applications/maas-controller-85c784d7b6-cnbhs to ip-10-0-1-23.us-east-2.compute.internal 2026/07/15 00:15:07 00:11:55 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 00:15:07 00:11:55 INFO Pod/maas-controller-85c784d7b6-cnbhs: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:5effee51faa067115d737a4aa57473635576fb8232b4ea3cc39f76c0146abf65" 2026/07/15 00:15:07 00:11:55 INFO Deployment/maas-controller: ScalingReplicaSet - Scaled up replica set maas-controller-85c784d7b6 from 0 to 1 2026/07/15 00:15:07 00:11:55 INFO Pod/maas-controller-85c784d7b6-cnbhs: AddedInterface - Add eth0 [10.130.0.158/23] from ovn-kubernetes 2026/07/15 00:15:07 00:11:55 INFO ReplicaSet/maas-controller-85c784d7b6: SuccessfulCreate - Created pod: maas-controller-85c784d7b6-cnbhs 2026/07/15 00:15:07 00:11:35 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-78dcbbdc6c-b5tgs_96c0874d-e87d-4922-9325-84bc43086b86 became leader 2026/07/15 00:15:07 00:11:33 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-7747874bf-gwjgq_1ff034d0-928c-4deb-ac2d-76d37c7f09ae became leader 2026/07/15 00:15:07 00:11:23 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-5b978c765f-npm2r_3a09efc7-17fe-4558-8691-a4913832d41f became leader 2026/07/15 00:15:07 00:11:18 INFO Pod/kserve-controller-manager-7747874bf-gwjgq: Created - Created container: manager 2026/07/15 00:15:07 00:11:18 INFO Pod/kserve-controller-manager-7747874bf-gwjgq: Started - Started container manager 2026/07/15 00:15:07 00:11:18 INFO Pod/kserve-controller-manager-7747874bf-gwjgq: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" in 522ms (522ms including waiting). Image size: 237680312 bytes. 2026/07/15 00:15:07 00:11:17 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-b5tgs: AddedInterface - Add eth0 [10.129.0.244/23] from ovn-kubernetes 2026/07/15 00:15:07 00:11:17 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-b5tgs: Started - Started container manager 2026/07/15 00:15:07 00:11:17 INFO Pod/kserve-controller-manager-7747874bf-gwjgq: AddedInterface - Add eth0 [10.130.0.157/23] from ovn-kubernetes 2026/07/15 00:15:07 00:11:17 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-b5tgs: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" 2026/07/15 00:15:07 00:11:17 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-b5tgs: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" in 500ms (500ms including waiting). Image size: 235181729 bytes. 2026/07/15 00:15:07 00:11:17 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-b5tgs: Created - Created container: manager 2026/07/15 00:15:07 00:11:17 INFO Pod/kserve-controller-manager-7747874bf-gwjgq: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" 2026/07/15 00:15:07 00:11:16 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-b5tgs: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-78dcbbdc6c-b5tgs to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 00:15:07 00:11:16 INFO Pod/kserve-controller-manager-7747874bf-gwjgq: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-7747874bf-gwjgq to ip-10-0-1-23.us-east-2.compute.internal 2026/07/15 00:15:07 00:11:16 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-78dcbbdc6c from 0 to 1 2026/07/15 00:15:07 00:11:16 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7747874bf from 0 to 1 2026/07/15 00:15:07 00:11:16 INFO ReplicaSet/kserve-controller-manager-7747874bf: SuccessfulCreate - Created pod: kserve-controller-manager-7747874bf-gwjgq 2026/07/15 00:15:07 00:11:16 INFO ReplicaSet/llmisvc-controller-manager-78dcbbdc6c: SuccessfulCreate - Created pod: llmisvc-controller-manager-78dcbbdc6c-b5tgs 2026/07/15 00:15:07 00:11:07 INFO Pod/model-serving-api-67c44bccdf-tzjzk: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:226370d82da697958a0cd64b5cfa82f290ae1bb701e74b72739fcac0b38a834d" already present on machine 2026/07/15 00:15:07 00:11:07 INFO Pod/model-serving-api-67c44bccdf-tzjzk: AddedInterface - Add eth0 [10.129.0.243/23] from ovn-kubernetes 2026/07/15 00:15:07 00:11:07 INFO Pod/model-serving-api-67c44bccdf-tzjzk: Created - Created container: server 2026/07/15 00:15:07 00:11:07 INFO Pod/model-serving-api-67c44bccdf-tzjzk: Started - Started container server 2026/07/15 00:15:07 00:11:06 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 00:15:07 00:11:06 INFO Pod/odh-model-controller-5b978c765f-npm2r: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" in 803ms (803ms including waiting). Image size: 238719886 bytes. 2026/07/15 00:15:07 00:11:06 INFO Pod/odh-model-controller-5b978c765f-npm2r: Created - Created container: manager 2026/07/15 00:15:07 00:11:06 INFO Pod/odh-model-controller-5b978c765f-npm2r: Started - Started container manager 2026/07/15 00:15:07 00:11:06 WARN Pod/model-serving-api-67c44bccdf-tzjzk: FailedMount - MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 2026/07/15 00:15:07 00:11:05 INFO Pod/model-serving-api-67c44bccdf-tzjzk: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-67c44bccdf-tzjzk to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 00:15:07 00:11:05 INFO Pod/odh-model-controller-5b978c765f-npm2r: AddedInterface - Add eth0 [10.129.0.242/23] from ovn-kubernetes 2026/07/15 00:15:07 00:11:05 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-67c44bccdf from 0 to 1 2026/07/15 00:15:07 00:11:05 INFO Pod/odh-model-controller-5b978c765f-npm2r: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" 2026/07/15 00:15:07 00:11:05 WARN ReplicaSet/model-serving-api-67c44bccdf: FailedCreate - Error creating: pods "model-serving-api-67c44bccdf-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/15 00:15:07 00:11:05 INFO ReplicaSet/model-serving-api-67c44bccdf: SuccessfulCreate - Created pod: model-serving-api-67c44bccdf-tzjzk 2026/07/15 00:15:07 00:11:04 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 00:15:07 00:11:04 INFO Pod/odh-model-controller-5b978c765f-npm2r: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-5b978c765f-npm2r to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 00:15:07 00:11:04 WARN Pod/odh-model-controller-5b978c765f-npm2r: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/15 00:15:07 00:11:04 INFO ReplicaSet/odh-model-controller-5b978c765f: SuccessfulCreate - Created pod: odh-model-controller-5b978c765f-npm2r 2026/07/15 00:15:07 00:11:04 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-5b978c765f from 0 to 1 2026/07/15 00:15:07 00:10:49 INFO Pod/maas-postgres-bb9d55498-4587g: Started - Started container postgres 2026/07/15 00:15:07 00:10:49 INFO Pod/maas-postgres-bb9d55498-4587g: Pulled - Successfully pulled image "registry.redhat.io/rhel9/postgresql-15:latest" in 5.302s (5.302s including waiting). Image size: 400815237 bytes. 2026/07/15 00:15:07 00:10:49 INFO Pod/maas-postgres-bb9d55498-4587g: Created - Created container: postgres 2026/07/15 00:15:07 00:10:44 INFO Pod/maas-postgres-bb9d55498-4587g: Pulling - Pulling image "registry.redhat.io/rhel9/postgresql-15:latest" 2026/07/15 00:15:07 00:10:44 INFO Pod/maas-postgres-bb9d55498-4587g: AddedInterface - Add eth0 [10.128.1.27/23] from ovn-kubernetes 2026/07/15 00:15:07 00:10:43 INFO Pod/maas-postgres-bb9d55498-4587g: Scheduled - Successfully assigned redhat-ods-applications/maas-postgres-bb9d55498-4587g to ip-10-0-1-62.us-east-2.compute.internal 2026/07/15 00:15:07 00:10:43 INFO Deployment/maas-postgres: ScalingReplicaSet - Scaled up replica set maas-postgres-bb9d55498 from 0 to 1 2026/07/15 00:15:07 00:10:43 INFO ReplicaSet/maas-postgres-bb9d55498: SuccessfulCreate - Created pod: maas-postgres-bb9d55498-4587g 2026/07/15 00:15:07 00:10:33 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-dxds2: Killing - Stopping container manager 2026/07/15 00:15:07 00:10:24 INFO Pod/model-serving-api-67c44bccdf-tnfnr: Killing - Stopping container server 2026/07/15 00:15:07 00:10:24 INFO Lease/b7e9931f.trustyai.opendatahub.io: LeaderElection - trustyai-service-operator-controller-manager-bc99cf85b-dxds2_2df69b54-dc50-4921-9924-08e46c89c754 became leader 2026/07/15 00:15:07 00:10:24 INFO Pod/odh-model-controller-5b978c765f-gzkdd: Killing - Stopping container manager 2026/07/15 00:15:07 00:10:24 INFO Pod/kserve-controller-manager-7747874bf-csnsz: Killing - Stopping container manager 2026/07/15 00:15:07 00:10:24 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-ldrnd: Killing - Stopping container manager 2026/07/15 00:15:07 00:10:08 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-dxds2: Scheduled - Successfully assigned redhat-ods-applications/trustyai-service-operator-controller-manager-bc99cf85b-dxds2 to ip-10-0-1-62.us-east-2.compute.internal 2026/07/15 00:15:07 00:10:08 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-csf6j: Killing - Stopping container manager 2026/07/15 00:15:07 00:10:08 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-dxds2: AddedInterface - Add eth0 [10.128.1.26/23] from ovn-kubernetes 2026/07/15 00:15:07 00:10:08 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-dxds2: Pulled - Container image "quay.io/rhoai/odh-trustyai-service-operator-rhel9@sha256:780ba93a8a4bf718fe19ad7bb9e4dec7b0ae79d5eb6d3e1e5060c3dc2260a515" already present on machine 2026/07/15 00:15:07 00:10:08 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-dxds2: Created - Created container: manager 2026/07/15 00:15:07 00:10:08 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-dxds2: Started - Started container manager 2026/07/15 00:15:07 00:10:08 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-7tj8z: Killing - Stopping container manager 2026/07/15 00:15:07 00:10:08 INFO ReplicaSet/trustyai-service-operator-controller-manager-bc99cf85b: SuccessfulCreate - Created pod: trustyai-service-operator-controller-manager-bc99cf85b-dxds2 2026/07/15 00:15:07 00:10:08 INFO Deployment/trustyai-service-operator-controller-manager: ScalingReplicaSet - Scaled up replica set trustyai-service-operator-controller-manager-bc99cf85b from 0 to 1 2026/07/15 00:15:07 === RESOURCE QUOTAS === 2026/07/15 00:15:07 No resource quotas found in monitored namespaces 2026/07/15 00:15:07 Classification for components: infrastructure/storage (code=1006, confidence=medium) 2026/07/15 00:15:07 Evidence: event Pod/kserve-controller-manager-7747874bf-tqwfg: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/kserve-controller-manager-7747874bf-tqwfg: FailedMount - MountVolume.SetUp failed for volume \"cert\" : secret \"kserve-webhook-server-cert\" not found"],"confidence":"medium"} 2026/07/15 00:15:07 Diagnostics complete for failed test: components controller_test.go:483: Stopping: components test failed. 2026/07/15 00:15:07 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/15 00:15:07 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/15 00:15:08 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (1469.82s)