=== RUN TestOdhOperator/components/group_2/kueue/Validate_component_removed_to_unmanaged_transition helper_test.go:150: Setting Kueue component (default-kueue) to Removed mode to start with a clean state. 2026-07-15T21:30:27Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.19.18"}} helper_test.go:150: Cleaning up any existing Kueue test resources. helper_test.go:150: Cleaning up Kueue cluster-scoped resources helper_test.go:150: Will attempt to delete 2 Kueue resources helper_test.go:150: Attempting to delete ClusterQueue /default helper_test.go:150: Successfully processed deletion of ClusterQueue /default helper_test.go:150: Attempting to delete Kueue /cluster helper_test.go:150: Successfully processed deletion of Kueue /cluster helper_test.go:150: Creating test namespace (test-kueue-managed-d9bvnsqkfsrs7b62482g) with Kueue management annotation. helper_test.go:150: Creating Kueue ConfigMap (kueue-manager-config) to verify default Kueue resource is created correctly. helper_test.go:150: Creating Kueue ConfigMap (kueue-manager-config) in namespace (redhat-ods-applications). helper_test.go:150: Updating the management state of the component in the DataScienceCluster to Unmanaged (expecting NotReady initially due to missing operator). helper_test.go:150: Ensuring LWS Operator is installed (namespace=openshift-lws-operator, name=leader-worker-set). helper_test.go:150: Ensuring LeaderWorkerSetOperator CR exists (namespace=openshift-lws-operator, name=cluster). helper_test.go:150: Ensuring OCP Kueue Operator is installed (namespace=openshift-kueue-operator, name=kueue-operator). helper_test.go:150: Verifying the component in the DataScienceCluster transitions to Ready state. helper_test.go:150: Verifying Kueue configuration (cluster) is created with all expected frameworks. testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_2/kueue/Validate_component_removed_to_unmanaged_transition (31.20s) === RUN TestOdhOperator/components/group_2/kueue === PAUSE TestOdhOperator/components/group_2/kueue === CONT TestOdhOperator/components/group_2/kueue test_context_test.go:223: Timed out after 10.001s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:114 with: Expected resource 'cluster' of kind 'Kueue' to match condition {(["BatchJob", "Deployment", "JobSet", "LeaderWorkerSet", "MPIJob", "PaddleJob", "Pod", "PyTorchJob", "RayCluster", "RayJob", "StatefulSet", "TFJob", "TrainJob", "XGBoostJob"] - .spec.config.integrations.frameworks) | length == 0 []}. Expected <string>: map[apiVersion:kueue.openshift.io/v1 kind:Kueue metadata:map[annotations:map[platform.opendatahub.io/instance.generation:1 platform.opendatahub.io/instance.name:default-kueue platform.opendatahub.io/instance.uid:2549494a-dbd6-4b97-b634-6c461640b7db platform.opendatahub.io/type:OpenShift AI Self-Managed platform.opendatahub.io/version:3.5.0-ea.2] creationTimestamp:2026-07-15T20:49:40Z deletionGracePeriodSeconds:0 deletionTimestamp:2026-07-15T21:30:38Z finalizers:[kueue.openshift.io/finalizer] generation:2 labels:map[platform.opendatahub.io/part-of:kueue] managedFields:[map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:status:map[f:conditions:map[k:{"type":"Available"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"CertManagerAvailable"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"Degraded"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"Progressing"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]]] f:readyReplicas:map[]]] manager:kueue-operator operation:Apply subresource:status time:2026-07-15T20:50:24Z] map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[.:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:labels:map[.:map[] f:platform.opendatahub.io/part-of:map[]]] f:spec:map[.:map[] f:config:map[.:map[] f:integrations:map[.:map[] f:frameworks:map[.:map[] v:"Deployment":map[] v:"Pod":map[] v:"PyTorchJob":map[] v:"RayCluster":map[] v:"RayJob":map[] v:"StatefulSet":map[] v:"TrainJob":map[]]]] f:logLevel:map[] f:managementState:map[] f:operatorLogLevel:map[]]] manager:manager operation:Update time:2026-07-15T20:49:40Z] map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:finalizers:map[.:map[] v:"kueue.openshift.io/finalizer":map[]]]] manager:kueue-operator operation:Update time:2026-07-15T21:30:37Z]] name:cluster resourceVersion:395237290 uid:8bb6c52d-5196-472e-a571-7a1ca1860781] spec:map[config:map[integrations:map[frameworks:[Deployment Pod PyTorchJob RayCluster RayJob StatefulSet TrainJob]]] logLevel:Normal managementState:Managed operatorLogLevel:Normal] status:map[conditions:[map[lastTransitionTime:2026-07-15T20:50:24Z message:All 2 replicas are ready reason:AllReplicasReady status:True type:Available] map[lastTransitionTime:2026-07-15T20:50:24Z message:Deployment is up to date reason:AsExpected status:False type:Progressing] map[lastTransitionTime:2026-07-15T20:50:24Z message: reason:AsExpected status:False type:Degraded] map[lastTransitionTime:2026-07-15T20:50:05Z message:cert-manager is installed reason:CertManagerInstalled status:True type:CertManagerAvailable]] readyReplicas:2]] to match expression <string>: (["BatchJob", "Deployment", "JobSet", "LeaderWorkerSet", "MPIJob", "PaddleJob", "Pod", "PyTorchJob", "RayCluster", "RayJob", "StatefulSet", "TFJob", "TrainJob", "XGBoostJob"] - .spec.config.integrations.frameworks) | length == 0 test_context_test.go:1106: cleanup: best-effort delete failed for /, Kind= redhat-ods-applications/kueue-manager-config: Object 'Kind' is missing in 'unstructured object has no kind' --- FAIL: TestOdhOperator/components/group_2/kueue (31.42s) === RUN TestOdhOperator/components/group_2 --- FAIL: TestOdhOperator/components/group_2 (0.00s) === RUN TestOdhOperator/components 2026/07/15 21:34:10 TEST FAILURE DETECTED: group 2 2026/07/15 21:34:10 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 2) === 2026/07/15 21:34:11 === CLUSTER STATE === 2026/07/15 21:34:11 Node: ods-qe-psi-07-b4mqp-master-0 2026/07/15 21:34:11 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 21:34:11 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 21:34:11 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 21:34:11 Condition Ready: True - kubelet is posting ready status 2026/07/15 21:34:11 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/15 21:34:11 Node: ods-qe-psi-07-b4mqp-master-1 2026/07/15 21:34:11 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 21:34:11 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 21:34:11 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 21:34:11 Condition Ready: True - kubelet is posting ready status 2026/07/15 21:34:11 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/15 21:34:11 Node: ods-qe-psi-07-b4mqp-master-2 2026/07/15 21:34:11 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 21:34:11 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 21:34:11 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 21:34:11 Condition Ready: True - kubelet is posting ready status 2026/07/15 21:34:11 Resources: 11500m CPU, 23450732Ki memory allocatable (of 12000m CPU, 24601708Ki memory capacity) 2026/07/15 21:34:11 Node: ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/15 21:34:11 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 21:34:11 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 21:34:11 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 21:34:11 Condition Ready: True - kubelet is posting ready status 2026/07/15 21:34:11 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/15 21:34:11 Node: ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/15 21:34:11 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 21:34:11 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 21:34:11 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 21:34:11 Condition Ready: True - kubelet is posting ready status 2026/07/15 21:34:11 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/15 21:34:11 Node: ods-qe-psi-07-b4mqp-worker-0-jd4pm 2026/07/15 21:34:11 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 21:34:11 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 21:34:11 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 21:34:11 Condition Ready: True - kubelet is posting ready status 2026/07/15 21:34:11 Resources: 11500m CPU, 23452736Ki memory allocatable (of 12000m CPU, 24603712Ki memory capacity) 2026/07/15 21:34:11 Node: ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/15 21:34:11 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 21:34:11 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 21:34:11 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 21:34:11 Condition Ready: True - kubelet is posting ready status 2026/07/15 21:34:11 Resources: 11500m CPU, 23452744Ki memory allocatable (of 12000m CPU, 24603720Ki memory capacity) 2026/07/15 21:34:11 Node: ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/15 21:34:11 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 21:34:11 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 21:34:11 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 21:34:11 Condition Ready: True - kubelet is posting ready status 2026/07/15 21:34:11 Resources: 11500m CPU, 23450720Ki memory allocatable (of 12000m CPU, 24601696Ki memory capacity) 2026/07/15 21:34:11 === DEPLOYMENTS === 2026/07/15 21:34:11 Namespace: redhat-ods-applications 2026/07/15 21:34:11 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/15 21:34:11 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/15 21:34:11 Namespace: kube-system 2026/07/15 21:34:11 === PODS === 2026/07/15 21:34:11 Namespace redhat-ods-applications: no problematic pods found 2026/07/15 21:34:11 Namespace kube-system: no problematic pods found 2026/07/15 21:34:11 === OPERATOR STATUS === 2026/07/15 21:34:11 Operator deployment: 3/3 ready 2026/07/15 21:34:11 === DSCI STATUS === 2026/07/15 21:34:11 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/15 21:34:11 === DSC STATUS === 2026/07/15 21:34:11 DSC default-dsc: 2026/07/15 21:34:11 Ready: True - 2026/07/15 21:34:11 ProvisioningSucceeded: True - 2026/07/15 21:34:11 ComponentsReady: True - 2026/07/15 21:34:11 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/15 21:34:11 DashboardReady: False - Component ManagementState is set to Removed 2026/07/15 21:34:11 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 21:34:11 KserveReady: False - Component ManagementState is set to Removed 2026/07/15 21:34:11 KueueReady: True - 2026/07/15 21:34:11 MLflowOperatorReady: True - 2026/07/15 21:34:11 ModelControllerReady: False - Component ManagementState is set to Removed 2026/07/15 21:34:11 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/15 21:34:11 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/15 21:34:11 OGXReady: False - Component ManagementState is set to Removed 2026/07/15 21:34:11 RayReady: False - Component ManagementState is set to Removed 2026/07/15 21:34:11 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 21:34:11 TrainerReady: False - Component ManagementState is set to Removed 2026/07/15 21:34:11 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 21:34:11 TrustyAIReady: True - 2026/07/15 21:34:11 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/15 21:34:11 === RECENT EVENTS === 2026/07/15 21:34:11 21:34:02 INFO Pod/kserve-controller-manager-6c67dcb69c-wfg6p: Killing - Stopping container manager 2026/07/15 21:34:11 21:34:02 INFO Pod/model-serving-api-6688cd56bd-ckvtb: Killing - Stopping container server 2026/07/15 21:34:11 21:34:02 INFO Pod/odh-model-controller-867855946f-vdqhl: Killing - Stopping container manager 2026/07/15 21:34:11 21:34:02 INFO Pod/model-registry-operator-controller-manager-76b5b7f674-lqx9w: Killing - Stopping container manager 2026/07/15 21:34:11 21:34:02 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-9wmkx: Killing - Stopping container manager 2026/07/15 21:34:11 21:34:01 INFO Pod/llmisvc-controller-manager-5d65df97bb-8qxzh: Killing - Stopping container manager 2026/07/15 21:34:11 21:33:48 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-867855946f-vdqhl_5c8e2951-f423-431d-91cd-f075b348c005 became leader 2026/07/15 21:34:11 21:33:46 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-5bd688895b-9wmkx_0598fa40-2a51-48bd-8673-dc3a45364f95 became leader 2026/07/15 21:34:11 21:33:45 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-9wmkx: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-5bd688895b-9wmkx to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/15 21:34:11 21:33:45 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-9wmkx: Created - Created container: manager 2026/07/15 21:34:11 21:33:45 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-5bd688895b-9wmkx 2026/07/15 21:34:11 21:33:45 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-5bd688895b from 0 to 1 2026/07/15 21:34:11 21:33:45 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-mx8tf: Killing - Stopping container manager 2026/07/15 21:34:11 21:33:45 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-9wmkx: Started - Started container manager 2026/07/15 21:34:11 21:33:45 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-4czb4: Killing - Stopping container manager 2026/07/15 21:34:11 21:33:45 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-9wmkx: AddedInterface - Add eth0 [10.130.3.25/23] from ovn-kubernetes 2026/07/15 21:34:11 21:33:45 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-9wmkx: Pulled - Container image "registry.redhat.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" already present on machine 2026/07/15 21:34:11 21:33:32 INFO Pod/odh-model-controller-867855946f-vdqhl: Started - Started container manager 2026/07/15 21:34:11 21:33:32 INFO Pod/odh-model-controller-867855946f-vdqhl: Created - Created container: manager 2026/07/15 21:34:11 21:33:32 INFO Pod/odh-model-controller-867855946f-vdqhl: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 1.467s (1.467s including waiting). Image size: 238959091 bytes. 2026/07/15 21:34:11 21:33:30 INFO Pod/odh-model-controller-867855946f-vdqhl: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-867855946f-vdqhl to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/15 21:34:11 21:33:30 INFO Pod/odh-model-controller-867855946f-vdqhl: AddedInterface - Add eth0 [10.130.3.24/23] from ovn-kubernetes 2026/07/15 21:34:11 21:33:30 INFO Pod/odh-model-controller-867855946f-vdqhl: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/15 21:34:11 21:33:30 INFO ReplicaSet/odh-model-controller-867855946f: SuccessfulCreate - Created pod: odh-model-controller-867855946f-vdqhl 2026/07/15 21:34:11 21:33:30 INFO Pod/odh-model-controller-867855946f-54bkp: Killing - Stopping container manager 2026/07/15 21:34:11 21:33:30 INFO Pod/odh-model-controller-867855946f-2bkgq: Killing - Stopping container manager 2026/07/15 21:34:11 21:33:30 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-867855946f from 0 to 1 2026/07/15 21:34:11 21:33:15 INFO Pod/model-serving-api-6688cd56bd-ckvtb: Created - Created container: server 2026/07/15 21:34:11 21:33:15 INFO Pod/model-serving-api-6688cd56bd-ckvtb: AddedInterface - Add eth0 [10.130.3.23/23] from ovn-kubernetes 2026/07/15 21:34:11 21:33:15 INFO Pod/model-serving-api-6688cd56bd-ckvtb: Started - Started container server 2026/07/15 21:34:11 21:33:15 INFO Pod/model-serving-api-6688cd56bd-ckvtb: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/15 21:34:11 21:33:14 INFO Pod/model-serving-api-6688cd56bd-ckvtb: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-ckvtb to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/15 21:34:11 21:33:14 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-ckvtb 2026/07/15 21:34:11 21:33:14 INFO Pod/model-serving-api-6688cd56bd-qds76: Killing - Stopping container server 2026/07/15 21:34:11 21:33:14 INFO Pod/model-serving-api-6688cd56bd-qg89x: Killing - Stopping container server 2026/07/15 21:34:11 21:33:14 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/15 21:34:11 21:32:16 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-mx8tf: Started - Started container manager 2026/07/15 21:34:11 21:32:16 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-mx8tf: Created - Created container: manager 2026/07/15 21:34:11 21:32:16 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-mx8tf: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" in 2.353s (2.353s including waiting). Image size: 195240093 bytes. 2026/07/15 21:34:11 21:32:14 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-mx8tf: Pulling - Pulling image "registry.redhat.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" 2026/07/15 21:34:11 21:32:14 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-mx8tf: AddedInterface - Add eth0 [10.131.1.165/23] from ovn-kubernetes 2026/07/15 21:34:11 21:32:13 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-mx8tf: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-5bd688895b-mx8tf to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/15 21:34:11 21:32:13 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-5bd688895b-mx8tf 2026/07/15 21:34:11 21:32:13 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-5bd688895b from 1 to 2 2026/07/15 21:34:11 21:31:55 INFO Pod/odh-model-controller-867855946f-54bkp: Started - Started container manager 2026/07/15 21:34:11 21:31:55 INFO Pod/odh-model-controller-867855946f-54bkp: Created - Created container: manager 2026/07/15 21:34:11 21:31:55 INFO Pod/odh-model-controller-867855946f-54bkp: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 632ms (632ms including waiting). Image size: 238959091 bytes. 2026/07/15 21:34:11 21:31:54 INFO Pod/odh-model-controller-867855946f-54bkp: AddedInterface - Add eth0 [10.130.3.22/23] from ovn-kubernetes 2026/07/15 21:34:11 21:31:54 INFO Pod/odh-model-controller-867855946f-54bkp: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/15 21:34:11 21:31:53 INFO Pod/odh-model-controller-867855946f-54bkp: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-867855946f-54bkp to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/15 21:34:11 21:31:53 INFO ReplicaSet/odh-model-controller-867855946f: SuccessfulCreate - Created pod: odh-model-controller-867855946f-54bkp 2026/07/15 21:34:11 21:31:53 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-867855946f from 1 to 2 2026/07/15 21:34:11 21:31:36 INFO Pod/model-serving-api-6688cd56bd-qg89x: Created - Created container: server 2026/07/15 21:34:11 21:31:36 INFO Pod/model-serving-api-6688cd56bd-qg89x: Started - Started container server 2026/07/15 21:34:11 21:31:36 INFO Pod/model-serving-api-6688cd56bd-qg89x: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" in 2.56s (2.56s including waiting). Image size: 172848429 bytes. 2026/07/15 21:34:11 21:31:34 INFO Pod/model-serving-api-6688cd56bd-qg89x: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" 2026/07/15 21:34:11 21:31:34 INFO Pod/model-serving-api-6688cd56bd-qg89x: AddedInterface - Add eth0 [10.131.1.164/23] from ovn-kubernetes 2026/07/15 21:34:11 21:31:33 INFO Pod/model-serving-api-6688cd56bd-qg89x: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-qg89x to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/15 21:34:11 21:31:33 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 1 to 2 2026/07/15 21:34:11 21:31:33 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-qg89x 2026/07/15 21:34:11 21:31:12 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 21:34:11 21:31:04 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-8qxzh_fc838d39-fb0f-4255-9b94-217ef237193d became leader 2026/07/15 21:34:11 21:31:01 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-wfg6p_e8ced50e-1f8a-471d-948a-15d636a8ceaf became leader 2026/07/15 21:34:11 21:30:55 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-5bd688895b-4czb4_8c1e187e-bd15-473f-8663-762b14dc857e became leader 2026/07/15 21:34:11 21:30:54 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-4czb4: Created - Created container: manager 2026/07/15 21:34:11 21:30:54 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-4czb4: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" in 2.493s (2.493s including waiting). Image size: 195240093 bytes. 2026/07/15 21:34:11 21:30:54 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-4czb4: Started - Started container manager 2026/07/15 21:34:11 21:30:52 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-4czb4: AddedInterface - Add eth0 [10.130.3.20/23] from ovn-kubernetes 2026/07/15 21:34:11 21:30:52 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-4czb4: Pulling - Pulling image "registry.redhat.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" 2026/07/15 21:34:11 21:30:51 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-4czb4: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-5bd688895b-4czb4 to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/15 21:34:11 21:30:51 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-5bd688895b-4czb4 2026/07/15 21:34:11 21:30:51 WARN Pod/workload-variant-autoscaler-controller-manager-5bd688895b-4czb4: FailedMount - MountVolume.SetUp failed for volume "epp-metrics-token" : secret "workload-variant-autoscaler-epp-metrics-token" not found 2026/07/15 21:34:11 21:30:51 WARN ServiceMonitor/workload-variant-autoscaler-controller-manager-metrics-monitor: InvalidConfiguration - ServiceMonitor workload-variant-autoscaler-controller-manager-metrics-monitor was rejected due to invalid configuration: unable to get secret "workload-variant-autoscaler-controller-manager-token": secrets "workload-variant-autoscaler-controller-manager-token" not found 2026/07/15 21:34:11 21:30:50 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-5bd688895b from 0 to 1 2026/07/15 21:34:11 21:30:50 WARN ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: FailedCreate - Error creating: pods "workload-variant-autoscaler-controller-manager-5bd688895b-" is forbidden: error looking up service account redhat-ods-applications/workload-variant-autoscaler-controller-manager: serviceaccount "workload-variant-autoscaler-controller-manager" not found 2026/07/15 21:34:11 21:30:48 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-867855946f-2bkgq_812d7340-865c-42c8-9996-af2ef1575961 became leader 2026/07/15 21:34:11 21:30:47 INFO Lease/85f368d1.opendatahub.io: LeaderElection - model-registry-operator-controller-manager-76b5b7f674-lqx9w_2e34c75d-e304-4b9f-9258-87e859f78d39 became leader 2026/07/15 21:34:11 21:30:46 INFO Pod/llmisvc-controller-manager-5d65df97bb-8qxzh: Created - Created container: manager 2026/07/15 21:34:11 21:30:46 INFO Pod/llmisvc-controller-manager-5d65df97bb-8qxzh: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 1.226s (1.226s including waiting). Image size: 236366599 bytes. 2026/07/15 21:34:11 21:30:46 INFO Pod/kserve-controller-manager-6c67dcb69c-wfg6p: Created - Created container: manager 2026/07/15 21:34:11 21:30:46 INFO Pod/kserve-controller-manager-6c67dcb69c-wfg6p: Started - Started container manager 2026/07/15 21:34:11 21:30:46 INFO Pod/llmisvc-controller-manager-5d65df97bb-8qxzh: Started - Started container manager 2026/07/15 21:34:11 21:30:46 INFO Pod/kserve-controller-manager-6c67dcb69c-wfg6p: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 1.207s (1.207s including waiting). Image size: 238435613 bytes. 2026/07/15 21:34:11 21:30:44 INFO Pod/llmisvc-controller-manager-5d65df97bb-8qxzh: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-8qxzh to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/15 21:34:11 21:30:44 INFO Pod/kserve-controller-manager-6c67dcb69c-wfg6p: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-wfg6p to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/15 21:34:11 21:30:44 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/15 21:34:11 21:30:44 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/15 21:34:11 21:30:44 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-8qxzh 2026/07/15 21:34:11 21:30:44 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-wfg6p 2026/07/15 21:34:11 21:30:44 INFO Pod/llmisvc-controller-manager-5d65df97bb-8qxzh: AddedInterface - Add eth0 [10.131.1.162/23] from ovn-kubernetes 2026/07/15 21:34:11 21:30:44 INFO Pod/llmisvc-controller-manager-5d65df97bb-8qxzh: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/15 21:34:11 21:30:44 INFO Pod/kserve-controller-manager-6c67dcb69c-wfg6p: AddedInterface - Add eth0 [10.129.3.132/23] from ovn-kubernetes 2026/07/15 21:34:11 21:30:44 INFO Pod/kserve-controller-manager-6c67dcb69c-wfg6p: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/15 21:34:11 21:30:32 INFO Pod/odh-model-controller-867855946f-2bkgq: Started - Started container manager 2026/07/15 21:34:11 21:30:32 INFO Pod/odh-model-controller-867855946f-2bkgq: Created - Created container: manager 2026/07/15 21:34:11 21:30:32 INFO Pod/odh-model-controller-867855946f-2bkgq: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 2.247s (2.247s including waiting). Image size: 238959091 bytes. 2026/07/15 21:34:11 21:30:31 INFO Pod/model-serving-api-6688cd56bd-qds76: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-qds76 to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/15 21:34:11 21:30:31 INFO Pod/model-serving-api-6688cd56bd-qds76: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/15 21:34:11 21:30:31 INFO Pod/model-serving-api-6688cd56bd-qds76: Started - Started container server 2026/07/15 21:34:11 21:30:31 INFO Pod/model-serving-api-6688cd56bd-qds76: Created - Created container: server 2026/07/15 21:34:11 21:30:31 INFO Pod/model-serving-api-6688cd56bd-qds76: AddedInterface - Add eth0 [10.130.3.19/23] from ovn-kubernetes 2026/07/15 21:34:11 21:30:30 INFO Pod/odh-model-controller-867855946f-2bkgq: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/15 21:34:11 21:30:30 INFO Pod/model-registry-operator-controller-manager-76b5b7f674-lqx9w: Created - Created container: manager 2026/07/15 21:34:11 21:30:30 INFO Pod/model-registry-operator-controller-manager-76b5b7f674-lqx9w: Started - Started container manager 2026/07/15 21:34:11 21:30:30 INFO Pod/model-registry-operator-controller-manager-76b5b7f674-lqx9w: Pulled - Container image "registry.redhat.io/rhoai/odh-model-registry-operator-rhel9@sha256:ebcec5c43468283ed3cbab50530f939166f8f3e8c9d306e91a8107d05d7fa9d4" already present on machine 2026/07/15 21:34:11 21:30:30 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/15 21:34:11 21:30:30 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-qds76 2026/07/15 21:34:11 21:30:30 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/15 21:34:11 21:30:30 INFO Pod/odh-model-controller-867855946f-2bkgq: AddedInterface - Add eth0 [10.129.3.131/23] from ovn-kubernetes 2026/07/15 21:34:11 21:30:30 INFO Pod/model-registry-operator-controller-manager-76b5b7f674-lqx9w: AddedInterface - Add eth0 [10.131.2.232/23] from ovn-kubernetes 2026/07/15 21:34:11 21:30:29 INFO Pod/odh-model-controller-867855946f-2bkgq: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-867855946f-2bkgq to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/15 21:34:11 21:30:29 INFO Pod/model-registry-operator-controller-manager-76b5b7f674-lqx9w: Scheduled - Successfully assigned redhat-ods-applications/model-registry-operator-controller-manager-76b5b7f674-lqx9w to ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/15 21:34:11 21:30:29 WARN Pod/odh-model-controller-867855946f-2bkgq: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/15 21:34:11 21:30:29 INFO Deployment/model-registry-operator-controller-manager: ScalingReplicaSet - Scaled up replica set model-registry-operator-controller-manager-76b5b7f674 from 0 to 1 2026/07/15 21:34:11 21:30:29 INFO ReplicaSet/odh-model-controller-867855946f: SuccessfulCreate - Created pod: odh-model-controller-867855946f-2bkgq 2026/07/15 21:34:11 21:30:29 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-867855946f from 0 to 1 2026/07/15 21:34:11 21:30:29 WARN Pod/model-registry-operator-controller-manager-76b5b7f674-lqx9w: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 2026/07/15 21:34:11 21:30:29 INFO ReplicaSet/model-registry-operator-controller-manager-76b5b7f674: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-76b5b7f674-lqx9w 2026/07/15 21:34:11 21:30:20 INFO Pod/dashboard-redirect-5bd755c985-krhvc: Killing - Stopping container nginx 2026/07/15 21:34:11 21:30:20 INFO Deployment/rhods-dashboard: ScalingReplicaSet - Scaled up replica set rhods-dashboard-d8f68f65 from 0 to 2 2026/07/15 21:34:11 21:30:20 WARN ReplicaSet/rhods-dashboard-d8f68f65: FailedCreate - Error creating: pods "rhods-dashboard-d8f68f65-" is forbidden: error looking up service account redhat-ods-applications/rhods-dashboard: serviceaccount "rhods-dashboard" not found 2026/07/15 21:34:11 21:30:20 INFO Pod/dashboard-redirect-5bd755c985-95zwz: Killing - Stopping container nginx 2026/07/15 21:34:11 21:30:18 WARN Pod/rhods-dashboard-d8f68f65-n6bxg: Unhealthy - Readiness probe failed: Get "https://10.131.1.161:8444/healthz": dial tcp 10.131.1.161:8444: connect: connection refused 2026/07/15 21:34:11 21:29:39 INFO Pod/spark-operator-webhook-9cb9bdf44-6r8kg: Killing - Stopping container webhook 2026/07/15 21:34:11 21:29:39 INFO Pod/spark-operator-controller-767899d44f-fr7gp: Killing - Stopping container controller 2026/07/15 21:34:11 21:29:39 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - ServiceMonitor controller-manager-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 21:34:11 21:29:33 INFO Pod/rhods-dashboard-d8f68f65-kd77h: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" in 2.633s (2.633s including waiting). Image size: 185092464 bytes. 2026/07/15 21:34:11 21:29:30 INFO Pod/rhods-dashboard-d8f68f65-kd77h: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" 2026/07/15 21:34:11 21:29:30 INFO Pod/rhods-dashboard-d8f68f65-kd77h: Started - Started container mlflow-ui 2026/07/15 21:34:11 21:29:30 INFO Pod/rhods-dashboard-d8f68f65-kd77h: Created - Created container: mlflow-ui 2026/07/15 21:34:11 21:29:30 INFO Pod/rhods-dashboard-d8f68f65-kd77h: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" in 2.537s (2.537s including waiting). Image size: 177808591 bytes. 2026/07/15 21:34:11 21:29:28 INFO Pod/rhods-dashboard-d8f68f65-kd77h: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" in 2.124s (2.124s including waiting). Image size: 187802205 bytes. 2026/07/15 21:34:11 21:29:28 INFO Pod/rhods-dashboard-d8f68f65-kd77h: Started - Started container maas-ui 2026/07/15 21:34:11 21:29:28 INFO Pod/rhods-dashboard-d8f68f65-kd77h: Created - Created container: maas-ui 2026/07/15 21:34:11 21:29:28 INFO Pod/rhods-dashboard-d8f68f65-kd77h: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" 2026/07/15 21:34:11 21:29:26 INFO Lease/spark-operator-controller-lock: LeaderElection - spark-operator-controller-767899d44f-fr7gp_c0e6023a-42c4-4abd-ba17-53bf813c9152 became leader 2026/07/15 21:34:11 21:29:26 INFO Pod/rhods-dashboard-d8f68f65-kd77h: Created - Created container: gen-ai-ui 2026/07/15 21:34:11 21:29:26 INFO Pod/rhods-dashboard-d8f68f65-kd77h: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" in 2.32s (2.32s including waiting). Image size: 241242949 bytes. 2026/07/15 21:34:11 21:29:26 INFO Pod/rhods-dashboard-d8f68f65-kd77h: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" 2026/07/15 21:34:11 21:29:26 INFO Pod/rhods-dashboard-d8f68f65-kd77h: Started - Started container gen-ai-ui 2026/07/15 21:34:11 21:29:24 INFO Pod/spark-operator-webhook-9cb9bdf44-6r8kg: Created - Created container: webhook 2026/07/15 21:34:11 21:29:24 INFO Pod/spark-operator-webhook-9cb9bdf44-6r8kg: AddedInterface - Add eth0 [10.130.3.16/23] from ovn-kubernetes 2026/07/15 21:34:11 21:29:24 INFO Pod/spark-operator-webhook-9cb9bdf44-6r8kg: Pulled - Container image "registry.redhat.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/15 21:34:11 21:29:24 INFO Pod/spark-operator-webhook-9cb9bdf44-6r8kg: Started - Started container webhook 2026/07/15 21:34:11 21:29:23 INFO Pod/spark-operator-webhook-9cb9bdf44-6r8kg: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-webhook-9cb9bdf44-6r8kg to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/15 21:34:11 21:29:23 INFO Pod/rhods-dashboard-d8f68f65-kd77h: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" in 1.796s (1.796s including waiting). Image size: 182438697 bytes. 2026/07/15 21:34:11 21:29:23 INFO Pod/spark-operator-webhook-9cb9bdf44-nc9tg: Killing - Stopping container webhook 2026/07/15 21:34:11 21:29:23 INFO Pod/spark-operator-webhook-9cb9bdf44-nd6jb: Killing - Stopping container webhook 2026/07/15 21:34:11 21:29:23 INFO ReplicaSet/spark-operator-webhook-9cb9bdf44: SuccessfulCreate - Created pod: spark-operator-webhook-9cb9bdf44-6r8kg 2026/07/15 21:34:11 21:29:23 INFO Deployment/spark-operator-webhook: ScalingReplicaSet - Scaled up replica set spark-operator-webhook-9cb9bdf44 from 0 to 1 2026/07/15 21:34:11 21:29:23 INFO Pod/rhods-dashboard-d8f68f65-kd77h: Created - Created container: model-registry-ui 2026/07/15 21:34:11 21:29:23 INFO Pod/rhods-dashboard-d8f68f65-kd77h: Started - Started container model-registry-ui 2026/07/15 21:34:11 21:29:23 INFO Pod/rhods-dashboard-d8f68f65-kd77h: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" 2026/07/15 21:34:11 21:29:21 INFO Pod/rhods-dashboard-d8f68f65-kd77h: Created - Created container: rhods-dashboard 2026/07/15 21:34:11 21:29:21 INFO Pod/rhods-dashboard-d8f68f65-kd77h: Pulled - Container image "registry.redhat.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:1e614e46799fd020ac18d95c06a463421154ce86e59ccb3fa6007e66fed9910e" already present on machine 2026/07/15 21:34:11 21:29:21 INFO Pod/rhods-dashboard-d8f68f65-kd77h: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" in 8.409s (8.409s including waiting). Image size: 799822033 bytes. 2026/07/15 21:34:11 21:29:21 INFO Pod/rhods-dashboard-d8f68f65-kd77h: Started - Started container rhods-dashboard 2026/07/15 21:34:11 21:29:21 INFO Pod/rhods-dashboard-d8f68f65-kd77h: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" 2026/07/15 21:34:11 21:29:21 INFO Pod/rhods-dashboard-d8f68f65-kd77h: Started - Started container kube-rbac-proxy 2026/07/15 21:34:11 21:29:21 INFO Pod/rhods-dashboard-d8f68f65-kd77h: Created - Created container: kube-rbac-proxy 2026/07/15 21:34:11 21:29:20 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" in 782ms (782ms including waiting). Image size: 185092464 bytes. 2026/07/15 21:34:11 21:29:19 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" 2026/07/15 21:34:11 21:29:19 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: Started - Started container mlflow-ui 2026/07/15 21:34:11 21:29:19 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: Created - Created container: mlflow-ui 2026/07/15 21:34:11 21:29:19 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" in 1.659s (1.659s including waiting). Image size: 177808591 bytes. 2026/07/15 21:34:11 21:29:17 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" 2026/07/15 21:34:11 21:29:17 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" 2026/07/15 21:34:11 21:29:17 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: Started - Started container gen-ai-ui 2026/07/15 21:34:11 21:29:17 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: Created - Created container: gen-ai-ui 2026/07/15 21:34:11 21:29:17 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" in 707ms (707ms including waiting). Image size: 187802205 bytes. 2026/07/15 21:34:11 21:29:17 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: Created - Created container: maas-ui 2026/07/15 21:34:11 21:29:17 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: Started - Started container maas-ui 2026/07/15 21:34:11 21:29:16 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" in 1.981s (1.982s including waiting). Image size: 241242949 bytes. 2026/07/15 21:34:11 21:29:14 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: Created - Created container: model-registry-ui 2026/07/15 21:34:11 21:29:14 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: Started - Started container model-registry-ui 2026/07/15 21:34:11 21:29:14 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: Started - Started container kube-rbac-proxy 2026/07/15 21:34:11 21:29:14 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: Created - Created container: kube-rbac-proxy 2026/07/15 21:34:11 21:29:14 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: Pulled - Container image "registry.redhat.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:1e614e46799fd020ac18d95c06a463421154ce86e59ccb3fa6007e66fed9910e" already present on machine 2026/07/15 21:34:11 21:29:14 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: Started - Started container rhods-dashboard 2026/07/15 21:34:11 21:29:14 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: Created - Created container: rhods-dashboard 2026/07/15 21:34:11 21:29:14 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" 2026/07/15 21:34:11 21:29:14 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" 2026/07/15 21:34:11 21:29:14 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" in 784ms (784ms including waiting). Image size: 182438697 bytes. 2026/07/15 21:34:11 21:29:13 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" in 1.089s (1.089s including waiting). Image size: 799822033 bytes. 2026/07/15 21:34:11 21:29:13 INFO Pod/rhods-dashboard-d8f68f65-kd77h: Pulling - Pulling image "registry.redhat.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" 2026/07/15 21:34:11 21:29:13 INFO Pod/rhods-dashboard-d8f68f65-kd77h: AddedInterface - Add eth0 [10.131.2.231/23] from ovn-kubernetes 2026/07/15 21:34:11 21:29:12 INFO Pod/rhods-dashboard-d8f68f65-kd77h: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-d8f68f65-kd77h to ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/15 21:34:11 21:29:12 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-d8f68f65-n6bxg to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/15 21:34:11 21:29:12 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: AddedInterface - Add eth0 [10.131.1.161/23] from ovn-kubernetes 2026/07/15 21:34:11 21:29:12 INFO Deployment/rhods-dashboard: ScalingReplicaSet - Scaled up replica set rhods-dashboard-d8f68f65 from 0 to 2 2026/07/15 21:34:11 21:29:12 INFO Pod/rhods-dashboard-d8f68f65-n6bxg: Pulling - Pulling image "registry.redhat.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" 2026/07/15 21:34:11 21:29:12 INFO ReplicaSet/rhods-dashboard-d8f68f65: SuccessfulCreate - Created pod: rhods-dashboard-d8f68f65-n6bxg 2026/07/15 21:34:11 21:29:12 INFO ReplicaSet/rhods-dashboard-d8f68f65: SuccessfulCreate - Created pod: rhods-dashboard-d8f68f65-kd77h 2026/07/15 21:34:11 === RESOURCE QUOTAS === 2026/07/15 21:34:11 No resource quotas found in monitored namespaces 2026/07/15 21:34:11 Classification for group 2: infrastructure/storage (code=1006, confidence=medium) 2026/07/15 21:34:11 Evidence: event Pod/workload-variant-autoscaler-controller-manager-5bd688895b-4czb4: FailedMount - MountVolume.SetUp failed for volume "epp-metrics-token" : secret "workload-variant-autoscaler-epp-metrics-token" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/workload-variant-autoscaler-controller-manager-5bd688895b-4czb4: FailedMount - MountVolume.SetUp failed for volume \"epp-metrics-token\" : secret \"workload-variant-autoscaler-epp-metrics-token\" not found"],"confidence":"medium"} 2026/07/15 21:34:11 Diagnostics complete for failed test: group 2 controller_test.go:753: Stopping: group 2 test failed. 2026/07/15 21:34:11 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/15 21:34:11 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/15 21:38:10 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (1061.63s) === RUN TestOdhOperator 2026/07/15 21:24:41 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/15 21:42:35 TEST FAILURE DETECTED: components 2026/07/15 21:42:35 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/15 21:42:35 === CLUSTER STATE === 2026/07/15 21:42:35 Node: ods-qe-psi-07-b4mqp-master-0 2026/07/15 21:42:35 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 21:42:35 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 21:42:35 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 21:42:35 Condition Ready: True - kubelet is posting ready status 2026/07/15 21:42:35 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/15 21:42:35 Node: ods-qe-psi-07-b4mqp-master-1 2026/07/15 21:42:35 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 21:42:35 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 21:42:35 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 21:42:35 Condition Ready: True - kubelet is posting ready status 2026/07/15 21:42:35 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/15 21:42:35 Node: ods-qe-psi-07-b4mqp-master-2 2026/07/15 21:42:35 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 21:42:35 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 21:42:35 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 21:42:35 Condition Ready: True - kubelet is posting ready status 2026/07/15 21:42:35 Resources: 11500m CPU, 23450732Ki memory allocatable (of 12000m CPU, 24601708Ki memory capacity) 2026/07/15 21:42:35 Node: ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/15 21:42:35 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 21:42:35 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 21:42:35 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 21:42:35 Condition Ready: True - kubelet is posting ready status 2026/07/15 21:42:35 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/15 21:42:35 Node: ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/15 21:42:35 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 21:42:35 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 21:42:35 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 21:42:35 Condition Ready: True - kubelet is posting ready status 2026/07/15 21:42:35 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/15 21:42:35 Node: ods-qe-psi-07-b4mqp-worker-0-jd4pm 2026/07/15 21:42:35 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 21:42:35 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 21:42:35 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 21:42:35 Condition Ready: True - kubelet is posting ready status 2026/07/15 21:42:35 Resources: 11500m CPU, 23452736Ki memory allocatable (of 12000m CPU, 24603712Ki memory capacity) 2026/07/15 21:42:35 Node: ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/15 21:42:35 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 21:42:35 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 21:42:35 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 21:42:35 Condition Ready: True - kubelet is posting ready status 2026/07/15 21:42:35 Resources: 11500m CPU, 23452744Ki memory allocatable (of 12000m CPU, 24603720Ki memory capacity) 2026/07/15 21:42:35 Node: ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/15 21:42:35 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 21:42:35 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 21:42:35 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 21:42:35 Condition Ready: True - kubelet is posting ready status 2026/07/15 21:42:35 Resources: 11500m CPU, 23450720Ki memory allocatable (of 12000m CPU, 24601696Ki memory capacity) 2026/07/15 21:42:35 === DEPLOYMENTS === 2026/07/15 21:42:35 Namespace: redhat-ods-applications 2026/07/15 21:42:35 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/15 21:42:35 Deployment kubeflow-trainer-controller-manager: 1/1 (READY) 2026/07/15 21:42:35 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/15 21:42:35 Deployment maas-postgres: 1/1 (READY) 2026/07/15 21:42:35 Deployment model-serving-api: 1/1 (READY) 2026/07/15 21:42:35 Deployment odh-model-controller: 1/1 (READY) 2026/07/15 21:42:35 Namespace: kube-system 2026/07/15 21:42:35 === PODS === 2026/07/15 21:42:35 Namespace redhat-ods-applications: no problematic pods found 2026/07/15 21:42:35 Namespace kube-system: no problematic pods found 2026/07/15 21:42:35 === OPERATOR STATUS === 2026/07/15 21:42:35 Operator deployment: 3/3 ready 2026/07/15 21:42:35 === DSCI STATUS === 2026/07/15 21:42:35 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/15 21:42:35 === DSC STATUS === 2026/07/15 21:42:35 DSC default-dsc: 2026/07/15 21:42:35 Ready: True - 2026/07/15 21:42:35 ProvisioningSucceeded: True - 2026/07/15 21:42:35 ComponentsReady: True - 2026/07/15 21:42:35 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/15 21:42:35 DashboardReady: False - Component ManagementState is set to Removed 2026/07/15 21:42:35 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 21:42:35 KserveLLMInferenceServiceDependencies: True - 2026/07/15 21:42:35 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/15 21:42:35 KserveReady: True - 2026/07/15 21:42:35 KueueReady: True - 2026/07/15 21:42:35 LLM-D-WVADependencies: True - 2026/07/15 21:42:35 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 21:42:35 ModelControllerReady: True - 2026/07/15 21:42:35 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/15 21:42:35 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/15 21:42:35 OGXReady: False - Component ManagementState is set to Removed 2026/07/15 21:42:35 RayReady: False - Component ManagementState is set to Removed 2026/07/15 21:42:35 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 21:42:35 TrainerReady: True - 2026/07/15 21:42:35 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 21:42:35 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/15 21:42:35 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/15 21:42:35 === RECENT EVENTS === 2026/07/15 21:42:35 21:42:17 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-58ld7_0c44c1cb-bd00-4a89-a34e-b59cc1b6be1a became leader 2026/07/15 21:42:35 21:42:17 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-thhx5_c71fe074-77d7-47e4-992a-77eef85f0bb3 became leader 2026/07/15 21:42:35 21:42:11 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 21:42:35 21:42:05 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-xwxjh_225a2082-e334-446f-9e09-125e52fb9d3a became leader 2026/07/15 21:42:35 21:42:01 INFO Pod/kserve-controller-manager-6c67dcb69c-58ld7: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 713ms (713ms including waiting). Image size: 238435613 bytes. 2026/07/15 21:42:35 21:42:01 INFO Pod/llmisvc-controller-manager-5d65df97bb-thhx5: Created - Created container: manager 2026/07/15 21:42:35 21:42:01 INFO Pod/llmisvc-controller-manager-5d65df97bb-thhx5: Started - Started container manager 2026/07/15 21:42:35 21:42:01 INFO Pod/kserve-controller-manager-6c67dcb69c-58ld7: Started - Started container manager 2026/07/15 21:42:35 21:42:01 INFO Pod/kserve-controller-manager-6c67dcb69c-58ld7: Created - Created container: manager 2026/07/15 21:42:35 21:42:00 INFO Pod/llmisvc-controller-manager-5d65df97bb-thhx5: AddedInterface - Add eth0 [10.131.2.244/23] from ovn-kubernetes 2026/07/15 21:42:35 21:42:00 INFO Pod/kserve-controller-manager-6c67dcb69c-58ld7: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/15 21:42:35 21:42:00 INFO Pod/kserve-controller-manager-6c67dcb69c-58ld7: AddedInterface - Add eth0 [10.131.1.174/23] from ovn-kubernetes 2026/07/15 21:42:35 21:42:00 INFO Pod/llmisvc-controller-manager-5d65df97bb-thhx5: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 767ms (767ms including waiting). Image size: 236366599 bytes. 2026/07/15 21:42:35 21:42:00 INFO Pod/llmisvc-controller-manager-5d65df97bb-thhx5: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/15 21:42:35 21:41:59 INFO Pod/llmisvc-controller-manager-5d65df97bb-thhx5: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-thhx5 to ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/15 21:42:35 21:41:59 INFO Pod/kserve-controller-manager-6c67dcb69c-58ld7: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-58ld7 to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/15 21:42:35 21:41:59 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-58ld7 2026/07/15 21:42:35 21:41:59 WARN Pod/kserve-controller-manager-6c67dcb69c-58ld7: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 2026/07/15 21:42:35 21:41:59 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/15 21:42:35 21:41:59 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-thhx5 2026/07/15 21:42:35 21:41:59 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/15 21:42:35 21:41:48 INFO Pod/model-serving-api-6688cd56bd-l4jmg: Started - Started container server 2026/07/15 21:42:35 21:41:48 INFO Pod/model-serving-api-6688cd56bd-l4jmg: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/15 21:42:35 21:41:48 INFO Pod/model-serving-api-6688cd56bd-l4jmg: AddedInterface - Add eth0 [10.130.3.35/23] from ovn-kubernetes 2026/07/15 21:42:35 21:41:48 INFO Pod/odh-model-controller-6849fc49d8-xwxjh: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 770ms (770ms including waiting). Image size: 238959091 bytes. 2026/07/15 21:42:35 21:41:48 INFO Pod/model-serving-api-6688cd56bd-l4jmg: Created - Created container: server 2026/07/15 21:42:35 21:41:48 INFO Pod/odh-model-controller-6849fc49d8-xwxjh: Created - Created container: manager 2026/07/15 21:42:35 21:41:48 INFO Pod/odh-model-controller-6849fc49d8-xwxjh: Started - Started container manager 2026/07/15 21:42:35 21:41:47 INFO Pod/model-serving-api-6688cd56bd-l4jmg: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-l4jmg to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/15 21:42:35 21:41:47 INFO Pod/odh-model-controller-6849fc49d8-xwxjh: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/15 21:42:35 21:41:47 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/15 21:42:35 21:41:47 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-l4jmg 2026/07/15 21:42:35 21:41:47 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/15 21:42:35 21:41:47 INFO Pod/odh-model-controller-6849fc49d8-xwxjh: AddedInterface - Add eth0 [10.131.1.173/23] from ovn-kubernetes 2026/07/15 21:42:35 21:41:46 INFO Pod/odh-model-controller-6849fc49d8-xwxjh: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-xwxjh to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/15 21:42:35 21:41:46 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-xwxjh 2026/07/15 21:42:35 21:41:46 WARN Pod/odh-model-controller-6849fc49d8-xwxjh: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/15 21:42:35 21:41:46 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/15 21:42:35 21:41:37 WARN ReplicaSet/kserve-controller-manager-6c67dcb69c: FailedCreate - Error creating: pods "kserve-controller-manager-6c67dcb69c-" is forbidden: error looking up service account redhat-ods-applications/kserve-controller-manager: serviceaccount "kserve-controller-manager" not found 2026/07/15 21:42:35 21:41:37 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/15 21:42:35 21:41:37 WARN ReplicaSet/llmisvc-controller-manager-5d65df97bb: FailedCreate - Error creating: pods "llmisvc-controller-manager-5d65df97bb-" is forbidden: error looking up service account redhat-ods-applications/llmisvc-controller-manager: serviceaccount "llmisvc-controller-manager" not found 2026/07/15 21:42:35 21:41:37 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/15 21:42:35 21:41:36 INFO Pod/odh-model-controller-6849fc49d8-4t6tf: Killing - Stopping container manager 2026/07/15 21:42:35 21:41:36 INFO Pod/model-serving-api-6688cd56bd-hb6dd: Killing - Stopping container server 2026/07/15 21:42:35 21:41:35 INFO Pod/llmisvc-controller-manager-5d65df97bb-24qf8: Killing - Stopping container manager 2026/07/15 21:42:35 21:41:35 INFO Pod/kserve-controller-manager-6c67dcb69c-t9q9l: Killing - Stopping container manager 2026/07/15 21:42:35 21:41:19 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-24qf8_74f24ff3-9b90-4461-b02e-019cd60b39ba became leader 2026/07/15 21:42:35 21:41:17 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-t9q9l_2c08b159-accc-481a-983d-46e2f76a7942 became leader 2026/07/15 21:42:35 21:41:03 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-4t6tf_7917e967-eaf2-4174-94b3-1ed6e27de5ea became leader 2026/07/15 21:42:35 21:41:01 INFO Pod/llmisvc-controller-manager-5d65df97bb-24qf8: Started - Started container manager 2026/07/15 21:42:35 21:41:01 INFO Pod/llmisvc-controller-manager-5d65df97bb-24qf8: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 870ms (870ms including waiting). Image size: 236366599 bytes. 2026/07/15 21:42:35 21:41:01 INFO Pod/kserve-controller-manager-6c67dcb69c-t9q9l: Started - Started container manager 2026/07/15 21:42:35 21:41:01 INFO Pod/kserve-controller-manager-6c67dcb69c-t9q9l: Created - Created container: manager 2026/07/15 21:42:35 21:41:01 INFO Pod/llmisvc-controller-manager-5d65df97bb-24qf8: Created - Created container: manager 2026/07/15 21:42:35 21:41:01 INFO Pod/kserve-controller-manager-6c67dcb69c-t9q9l: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 665ms (665ms including waiting). Image size: 238435613 bytes. 2026/07/15 21:42:35 21:41:00 INFO Pod/llmisvc-controller-manager-5d65df97bb-24qf8: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/15 21:42:35 21:41:00 INFO Pod/llmisvc-controller-manager-5d65df97bb-24qf8: AddedInterface - Add eth0 [10.131.2.243/23] from ovn-kubernetes 2026/07/15 21:42:35 21:41:00 INFO Pod/kserve-controller-manager-6c67dcb69c-t9q9l: AddedInterface - Add eth0 [10.131.1.172/23] from ovn-kubernetes 2026/07/15 21:42:35 21:41:00 INFO Pod/kserve-controller-manager-6c67dcb69c-t9q9l: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/15 21:42:35 21:40:59 INFO Pod/llmisvc-controller-manager-5d65df97bb-24qf8: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-24qf8 to ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/15 21:42:35 21:40:59 INFO Pod/kserve-controller-manager-6c67dcb69c-t9q9l: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-t9q9l to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/15 21:42:35 21:40:59 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/15 21:42:35 21:40:59 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/15 21:42:35 21:40:59 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-t9q9l 2026/07/15 21:42:35 21:40:59 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-24qf8 2026/07/15 21:42:35 21:40:56 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-7ccfdd66-j92xv_30c72015-c7a2-4038-8a61-de7208274d78 became leader 2026/07/15 21:42:35 21:40:47 INFO Pod/odh-model-controller-6849fc49d8-4t6tf: Started - Started container manager 2026/07/15 21:42:35 21:40:47 INFO Pod/odh-model-controller-6849fc49d8-4t6tf: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 1.285s (1.285s including waiting). Image size: 238959091 bytes. 2026/07/15 21:42:35 21:40:47 INFO Pod/odh-model-controller-6849fc49d8-4t6tf: Created - Created container: manager 2026/07/15 21:42:35 21:40:46 INFO Pod/odh-model-controller-6849fc49d8-4t6tf: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/15 21:42:35 21:40:46 INFO Pod/model-serving-api-6688cd56bd-hb6dd: Created - Created container: server 2026/07/15 21:42:35 21:40:46 INFO Pod/model-serving-api-6688cd56bd-hb6dd: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/15 21:42:35 21:40:46 INFO Pod/model-serving-api-6688cd56bd-hb6dd: AddedInterface - Add eth0 [10.131.2.242/23] from ovn-kubernetes 2026/07/15 21:42:35 21:40:46 INFO Pod/odh-model-controller-6849fc49d8-4t6tf: AddedInterface - Add eth0 [10.130.3.34/23] from ovn-kubernetes 2026/07/15 21:42:35 21:40:46 INFO Pod/model-serving-api-6688cd56bd-hb6dd: Started - Started container server 2026/07/15 21:42:35 21:40:45 INFO Pod/model-serving-api-6688cd56bd-hb6dd: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-hb6dd to ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/15 21:42:35 21:40:45 INFO Pod/odh-model-controller-6849fc49d8-4t6tf: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-4t6tf to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/15 21:42:35 21:40:45 WARN Pod/odh-model-controller-6849fc49d8-4t6tf: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/15 21:42:35 21:40:45 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-hb6dd 2026/07/15 21:42:35 21:40:45 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-4t6tf 2026/07/15 21:42:35 21:40:45 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/15 21:42:35 21:40:45 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 21:42:35 21:40:45 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/15 21:42:35 21:40:45 WARN Pod/model-serving-api-6688cd56bd-hb6dd: FailedMount - MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 2026/07/15 21:42:35 21:40:43 INFO Pod/kserve-controller-manager-6c67dcb69c-s98jx: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 2.664s (2.664s including waiting). Image size: 238435613 bytes. 2026/07/15 21:42:35 21:40:43 INFO Pod/kserve-controller-manager-6c67dcb69c-s98jx: Created - Created container: manager 2026/07/15 21:42:35 21:40:43 INFO Pod/kserve-controller-manager-6c67dcb69c-s98jx: Started - Started container manager 2026/07/15 21:42:35 21:40:41 INFO Pod/kserve-controller-manager-6c67dcb69c-s98jx: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/15 21:42:35 21:40:41 INFO Pod/kserve-controller-manager-6c67dcb69c-s98jx: AddedInterface - Add eth0 [10.131.1.171/23] from ovn-kubernetes 2026/07/15 21:42:35 21:40:40 INFO Pod/kserve-controller-manager-6c67dcb69c-s98jx: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-s98jx to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/15 21:42:35 21:40:40 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-s98jx 2026/07/15 21:42:35 21:40:40 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/15 21:42:35 21:40:39 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-j92xv: Started - Started container manager 2026/07/15 21:42:35 21:40:39 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-j92xv: AddedInterface - Add eth0 [10.130.3.33/23] from ovn-kubernetes 2026/07/15 21:42:35 21:40:39 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-j92xv: Pulled - Container image "registry.redhat.io/rhoai/odh-trainer-rhel9@sha256:59ea523e2df9c1e705c755308ac11420682081cc63d54890e2b15fbd7d8ba570" already present on machine 2026/07/15 21:42:35 21:40:39 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-j92xv: Created - Created container: manager 2026/07/15 21:42:35 21:40:38 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-j92xv: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-7ccfdd66-j92xv to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/15 21:42:35 21:40:38 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/15 21:42:35 21:40:38 INFO ReplicaSet/kubeflow-trainer-controller-manager-7ccfdd66: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-7ccfdd66-j92xv 2026/07/15 21:42:35 21:40:38 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/15 21:42:35 21:40:38 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-7ccfdd66 from 0 to 1 2026/07/15 21:42:35 21:40:37 WARN ReplicaSet/odh-model-controller-6849fc49d8: FailedCreate - Error creating: pods "odh-model-controller-6849fc49d8-" is forbidden: error looking up service account redhat-ods-applications/odh-model-controller: serviceaccount "odh-model-controller" not found 2026/07/15 21:42:35 21:40:37 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 21:42:35 21:40:36 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/15 21:42:35 21:40:35 INFO Pod/model-serving-api-6688cd56bd-g5bqj: Killing - Stopping container server 2026/07/15 21:42:35 21:40:35 INFO Pod/kserve-controller-manager-6c67dcb69c-2k78w: Killing - Stopping container manager 2026/07/15 21:42:35 21:40:35 INFO Pod/odh-model-controller-6849fc49d8-wk6qg: Killing - Stopping container manager 2026/07/15 21:42:35 21:40:35 INFO Pod/llmisvc-controller-manager-5d65df97bb-9l6kw: Killing - Stopping container manager 2026/07/15 21:42:35 21:40:35 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 21:42:35 21:40:23 INFO Pod/maas-api-84449bdd5d-rxf4k: Killing - Stopping container maas-api 2026/07/15 21:42:35 21:40:23 INFO Pod/maas-controller-85bdcf9988-zcfvt: Killing - Stopping container manager 2026/07/15 21:42:35 21:40:04 INFO Pod/maas-api-84449bdd5d-rxf4k: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" in 4.902s (4.903s including waiting). Image size: 175018396 bytes. 2026/07/15 21:42:35 21:40:04 INFO Pod/maas-api-84449bdd5d-rxf4k: Created - Created container: maas-api 2026/07/15 21:42:35 21:40:04 INFO Pod/maas-api-84449bdd5d-rxf4k: Started - Started container maas-api 2026/07/15 21:42:35 21:39:59 INFO Pod/maas-api-84449bdd5d-rxf4k: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" 2026/07/15 21:42:35 21:39:59 INFO Pod/maas-api-84449bdd5d-rxf4k: AddedInterface - Add eth0 [10.130.3.31/23] from ovn-kubernetes 2026/07/15 21:42:35 21:39:58 INFO Pod/maas-api-84449bdd5d-rxf4k: Scheduled - Successfully assigned redhat-ods-applications/maas-api-84449bdd5d-rxf4k to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/15 21:42:35 21:39:58 INFO Deployment/maas-api: ScalingReplicaSet - Scaled up replica set maas-api-84449bdd5d from 0 to 1 2026/07/15 21:42:35 21:39:58 WARN ReplicaSet/maas-api-84449bdd5d: FailedCreate - Error creating: pods "maas-api-84449bdd5d-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 2026/07/15 21:42:35 21:39:58 INFO ReplicaSet/maas-api-84449bdd5d: SuccessfulCreate - Created pod: maas-api-84449bdd5d-rxf4k 2026/07/15 21:42:35 21:39:26 INFO Pod/maas-controller-85bdcf9988-zcfvt: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 2.082s (2.082s including waiting). Image size: 212866077 bytes. 2026/07/15 21:42:35 21:39:26 INFO Lease/maas-controller.models-as-a-service.opendatahub.io: LeaderElection - maas-controller-85bdcf9988-zcfvt_f20a41b2-fd08-4e57-b31d-b1f291590a39 became leader 2026/07/15 21:42:35 21:39:26 INFO Pod/maas-controller-85bdcf9988-zcfvt: Started - Started container manager 2026/07/15 21:42:35 21:39:26 INFO Pod/maas-controller-85bdcf9988-zcfvt: Created - Created container: manager 2026/07/15 21:42:35 21:39:24 INFO Pod/maas-controller-85bdcf9988-zcfvt: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/15 21:42:35 21:39:24 INFO Pod/maas-controller-85bdcf9988-zcfvt: AddedInterface - Add eth0 [10.131.1.170/23] from ovn-kubernetes 2026/07/15 21:42:35 21:39:23 INFO Pod/maas-controller-85bdcf9988-zcfvt: Scheduled - Successfully assigned redhat-ods-applications/maas-controller-85bdcf9988-zcfvt to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/15 21:42:35 21:39:23 INFO Deployment/maas-controller: ScalingReplicaSet - Scaled up replica set maas-controller-85bdcf9988 from 0 to 1 2026/07/15 21:42:35 21:39:23 INFO ReplicaSet/maas-controller-85bdcf9988: SuccessfulCreate - Created pod: maas-controller-85bdcf9988-zcfvt 2026/07/15 21:42:35 21:39:03 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-2k78w_c154e9b4-e488-4a4a-9484-8590e0777edd became leader 2026/07/15 21:42:35 21:39:03 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-9l6kw_b51aa447-ad2e-4f01-afaf-dbcde184dd52 became leader 2026/07/15 21:42:35 21:38:53 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-wk6qg_078ddac9-6686-416a-b097-8bb96afeaa5c became leader 2026/07/15 21:42:35 21:38:46 INFO Pod/kserve-controller-manager-6c67dcb69c-2k78w: Created - Created container: manager 2026/07/15 21:42:35 21:38:46 INFO Pod/kserve-controller-manager-6c67dcb69c-2k78w: Started - Started container manager 2026/07/15 21:42:35 21:38:46 INFO Pod/kserve-controller-manager-6c67dcb69c-2k78w: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 1.116s (1.116s including waiting). Image size: 238435613 bytes. 2026/07/15 21:42:35 21:38:46 INFO Pod/llmisvc-controller-manager-5d65df97bb-9l6kw: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 933ms (933ms including waiting). Image size: 236366599 bytes. 2026/07/15 21:42:35 21:38:46 INFO Pod/llmisvc-controller-manager-5d65df97bb-9l6kw: Created - Created container: manager 2026/07/15 21:42:35 21:38:46 INFO Pod/llmisvc-controller-manager-5d65df97bb-9l6kw: Started - Started container manager 2026/07/15 21:42:35 21:38:45 INFO Pod/llmisvc-controller-manager-5d65df97bb-9l6kw: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-9l6kw to ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/15 21:42:35 21:38:45 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-9l6kw 2026/07/15 21:42:35 21:38:45 INFO Pod/kserve-controller-manager-6c67dcb69c-2k78w: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/15 21:42:35 21:38:45 INFO Pod/llmisvc-controller-manager-5d65df97bb-9l6kw: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/15 21:42:35 21:38:45 INFO Pod/llmisvc-controller-manager-5d65df97bb-9l6kw: AddedInterface - Add eth0 [10.131.2.240/23] from ovn-kubernetes 2026/07/15 21:42:35 21:38:45 INFO Pod/kserve-controller-manager-6c67dcb69c-2k78w: AddedInterface - Add eth0 [10.130.3.30/23] from ovn-kubernetes 2026/07/15 21:42:35 21:38:44 INFO Pod/kserve-controller-manager-6c67dcb69c-2k78w: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-2k78w to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/15 21:42:35 21:38:44 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/15 21:42:35 21:38:44 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-2k78w 2026/07/15 21:42:35 21:38:44 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/15 21:42:35 21:38:35 INFO Pod/odh-model-controller-6849fc49d8-wk6qg: Created - Created container: manager 2026/07/15 21:42:35 21:38:35 INFO Pod/odh-model-controller-6849fc49d8-wk6qg: Started - Started container manager 2026/07/15 21:42:35 21:38:35 INFO Pod/odh-model-controller-6849fc49d8-wk6qg: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 1.552s (1.552s including waiting). Image size: 238959091 bytes. 2026/07/15 21:42:35 21:38:34 INFO Pod/model-serving-api-6688cd56bd-g5bqj: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-g5bqj to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/15 21:42:35 21:38:34 INFO Pod/odh-model-controller-6849fc49d8-wk6qg: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/15 21:42:35 21:38:34 INFO Pod/odh-model-controller-6849fc49d8-wk6qg: AddedInterface - Add eth0 [10.130.3.29/23] from ovn-kubernetes 2026/07/15 21:42:35 21:38:34 INFO Pod/model-serving-api-6688cd56bd-g5bqj: Created - Created container: server 2026/07/15 21:42:35 21:38:34 INFO Pod/model-serving-api-6688cd56bd-g5bqj: Started - Started container server 2026/07/15 21:42:35 21:38:34 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-g5bqj 2026/07/15 21:42:35 21:38:34 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/15 21:42:35 21:38:34 INFO Pod/model-serving-api-6688cd56bd-g5bqj: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/15 21:42:35 21:38:34 INFO Pod/model-serving-api-6688cd56bd-g5bqj: AddedInterface - Add eth0 [10.131.1.169/23] from ovn-kubernetes 2026/07/15 21:42:35 21:38:33 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/15 21:42:35 21:38:33 WARN Pod/odh-model-controller-6849fc49d8-wk6qg: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/15 21:42:35 21:38:32 INFO Pod/odh-model-controller-6849fc49d8-wk6qg: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-wk6qg to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/15 21:42:35 21:38:32 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/15 21:42:35 21:38:32 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-wk6qg 2026/07/15 21:42:35 21:38:17 INFO Pod/maas-postgres-d66f546cf-2kmfr: Started - Started container postgres 2026/07/15 21:42:35 21:38:17 INFO Pod/maas-postgres-d66f546cf-2kmfr: Pulled - Successfully pulled image "registry.redhat.io/rhel9/postgresql-15:latest" in 5.607s (5.607s including waiting). Image size: 400815237 bytes. 2026/07/15 21:42:35 21:38:17 INFO Pod/maas-postgres-d66f546cf-2kmfr: Created - Created container: postgres 2026/07/15 21:42:35 21:38:11 INFO Pod/maas-postgres-d66f546cf-2kmfr: Scheduled - Successfully assigned redhat-ods-applications/maas-postgres-d66f546cf-2kmfr to ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/15 21:42:35 21:38:11 INFO Pod/maas-postgres-d66f546cf-2kmfr: AddedInterface - Add eth0 [10.131.2.239/23] from ovn-kubernetes 2026/07/15 21:42:35 21:38:11 INFO ReplicaSet/maas-postgres-d66f546cf: SuccessfulCreate - Created pod: maas-postgres-d66f546cf-2kmfr 2026/07/15 21:42:35 21:38:11 INFO Pod/maas-postgres-d66f546cf-2kmfr: Pulling - Pulling image "registry.redhat.io/rhel9/postgresql-15:latest" 2026/07/15 21:42:35 21:38:11 INFO Deployment/maas-postgres: ScalingReplicaSet - Scaled up replica set maas-postgres-d66f546cf from 0 to 1 2026/07/15 21:42:35 21:38:00 INFO Pod/trustyai-service-operator-controller-manager-85df4c5659-tcmg7: Killing - Stopping container manager 2026/07/15 21:42:35 21:37:52 INFO Lease/b7e9931f.trustyai.opendatahub.io: LeaderElection - trustyai-service-operator-controller-manager-85df4c5659-tcmg7_30a7e36b-a844-4164-9452-4a7d32c33c33 became leader 2026/07/15 21:42:35 21:37:50 INFO Pod/kserve-controller-manager-6c67dcb69c-z6q9m: Killing - Stopping container manager 2026/07/15 21:42:35 21:37:50 INFO Pod/odh-model-controller-6849fc49d8-qqhnb: Killing - Stopping container manager 2026/07/15 21:42:35 21:37:50 INFO Pod/llmisvc-controller-manager-5d65df97bb-6fqvz: Killing - Stopping container manager 2026/07/15 21:42:35 21:37:50 INFO Pod/model-serving-api-6688cd56bd-pgjgg: Killing - Stopping container server 2026/07/15 21:42:35 21:37:50 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 21:42:35 === RESOURCE QUOTAS === 2026/07/15 21:42:35 No resource quotas found in monitored namespaces 2026/07/15 21:42:35 Classification for components: infrastructure/storage (code=1006, confidence=medium) 2026/07/15 21:42:35 Evidence: event Pod/kserve-controller-manager-6c67dcb69c-58ld7: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/kserve-controller-manager-6c67dcb69c-58ld7: FailedMount - MountVolume.SetUp failed for volume \"cert\" : secret \"kserve-webhook-server-cert\" not found"],"confidence":"medium"} 2026/07/15 21:42:35 Diagnostics complete for failed test: components controller_test.go:484: Stopping: components test failed. 2026/07/15 21:42:35 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/15 21:42:35 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/15 21:42:37 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (1075.44s)