=== RUN TestOdhOperator/components/group_2/kueue/Validate_component_removed_to_unmanaged_transition helper_test.go:150: Setting Kueue component (default-kueue) to Removed mode to start with a clean state. 2026-07-16T15:51:17Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.19.18"}} helper_test.go:150: Cleaning up any existing Kueue test resources. helper_test.go:150: Cleaning up Kueue cluster-scoped resources helper_test.go:150: Will attempt to delete 2 Kueue resources helper_test.go:150: Attempting to delete ClusterQueue /default helper_test.go:150: Successfully processed deletion of ClusterQueue /default helper_test.go:150: Attempting to delete Kueue /cluster helper_test.go:150: Successfully processed deletion of Kueue /cluster helper_test.go:150: Creating test namespace (test-kueue-managed-d9cfrtfs732s7b3o127g) with Kueue management annotation. helper_test.go:150: Creating Kueue ConfigMap (kueue-manager-config) to verify default Kueue resource is created correctly. helper_test.go:150: Creating Kueue ConfigMap (kueue-manager-config) in namespace (redhat-ods-applications). helper_test.go:150: Updating the management state of the component in the DataScienceCluster to Unmanaged (expecting NotReady initially due to missing operator). helper_test.go:150: Ensuring LWS Operator is installed (namespace=openshift-lws-operator, name=leader-worker-set). helper_test.go:150: Ensuring LeaderWorkerSetOperator CR exists (namespace=openshift-lws-operator, name=cluster). helper_test.go:150: Ensuring OCP Kueue Operator is installed (namespace=openshift-kueue-operator, name=kueue-operator). helper_test.go:150: Verifying the component in the DataScienceCluster transitions to Ready state. helper_test.go:150: Verifying Kueue configuration (cluster) is created with all expected frameworks. testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_2/kueue/Validate_component_removed_to_unmanaged_transition (30.67s) === RUN TestOdhOperator/components/group_2/kueue === PAUSE TestOdhOperator/components/group_2/kueue === CONT TestOdhOperator/components/group_2/kueue test_context_test.go:223: Timed out after 10.001s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:114 with: Expected resource 'cluster' of kind 'Kueue' to match condition {(["BatchJob", "Deployment", "JobSet", "LeaderWorkerSet", "MPIJob", "PaddleJob", "Pod", "PyTorchJob", "RayCluster", "RayJob", "StatefulSet", "TFJob", "TrainJob", "XGBoostJob"] - .spec.config.integrations.frameworks) | length == 0 []}. Expected <string>: map[apiVersion:kueue.openshift.io/v1 kind:Kueue metadata:map[annotations:map[platform.opendatahub.io/instance.generation:1 platform.opendatahub.io/instance.name:default-kueue platform.opendatahub.io/instance.uid:22a03c5b-fcbc-4c7d-bea7-2f2d772ac304 platform.opendatahub.io/type:OpenShift AI Self-Managed platform.opendatahub.io/version:3.5.0-ea.2] creationTimestamp:2026-07-16T15:03:12Z deletionGracePeriodSeconds:0 deletionTimestamp:2026-07-16T15:51:27Z finalizers:[kueue.openshift.io/finalizer] generation:2 labels:map[platform.opendatahub.io/part-of:kueue] managedFields:[map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:status:map[f:conditions:map[k:{"type":"Available"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"CertManagerAvailable"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"Degraded"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"Progressing"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]]] f:readyReplicas:map[]]] manager:kueue-operator operation:Apply subresource:status time:2026-07-16T15:03:53Z] map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[.:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:labels:map[.:map[] f:platform.opendatahub.io/part-of:map[]]] f:spec:map[.:map[] f:config:map[.:map[] f:integrations:map[.:map[] f:frameworks:map[.:map[] v:"Deployment":map[] v:"Pod":map[] v:"PyTorchJob":map[] v:"RayCluster":map[] v:"RayJob":map[] v:"StatefulSet":map[] v:"TrainJob":map[]]]] f:logLevel:map[] f:managementState:map[] f:operatorLogLevel:map[]]] manager:manager operation:Update time:2026-07-16T15:03:12Z] map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:finalizers:map[.:map[] v:"kueue.openshift.io/finalizer":map[]]]] manager:kueue-operator operation:Update time:2026-07-16T15:51:27Z]] name:cluster resourceVersion:396158144 uid:f99b7efa-f546-4162-b209-be9010b85a92] spec:map[config:map[integrations:map[frameworks:[Deployment Pod PyTorchJob RayCluster RayJob StatefulSet TrainJob]]] logLevel:Normal managementState:Managed operatorLogLevel:Normal] status:map[conditions:[map[lastTransitionTime:2026-07-16T15:03:53Z message:All 2 replicas are ready reason:AllReplicasReady status:True type:Available] map[lastTransitionTime:2026-07-16T15:03:53Z message:Deployment is up to date reason:AsExpected status:False type:Progressing] map[lastTransitionTime:2026-07-16T15:03:53Z message: reason:AsExpected status:False type:Degraded] map[lastTransitionTime:2026-07-16T15:03:37Z message:cert-manager is installed reason:CertManagerInstalled status:True type:CertManagerAvailable]] readyReplicas:2]] to match expression <string>: (["BatchJob", "Deployment", "JobSet", "LeaderWorkerSet", "MPIJob", "PaddleJob", "Pod", "PyTorchJob", "RayCluster", "RayJob", "StatefulSet", "TFJob", "TrainJob", "XGBoostJob"] - .spec.config.integrations.frameworks) | length == 0 test_context_test.go:1106: cleanup: best-effort delete failed for /, Kind= redhat-ods-applications/kueue-manager-config: Object 'Kind' is missing in 'unstructured object has no kind' --- FAIL: TestOdhOperator/components/group_2/kueue (30.85s) === RUN TestOdhOperator/components/group_2 --- FAIL: TestOdhOperator/components/group_2 (0.00s) === RUN TestOdhOperator/components 2026/07/16 15:54:59 TEST FAILURE DETECTED: group 2 2026/07/16 15:54:59 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 2) === 2026/07/16 15:54:59 === CLUSTER STATE === 2026/07/16 15:54:59 Node: ods-qe-psi-07-b4mqp-master-0 2026/07/16 15:54:59 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 15:54:59 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 15:54:59 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 15:54:59 Condition Ready: True - kubelet is posting ready status 2026/07/16 15:54:59 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/16 15:54:59 Node: ods-qe-psi-07-b4mqp-master-1 2026/07/16 15:54:59 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 15:54:59 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 15:54:59 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 15:54:59 Condition Ready: True - kubelet is posting ready status 2026/07/16 15:54:59 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/16 15:54:59 Node: ods-qe-psi-07-b4mqp-master-2 2026/07/16 15:54:59 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 15:54:59 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 15:54:59 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 15:54:59 Condition Ready: True - kubelet is posting ready status 2026/07/16 15:54:59 Resources: 11500m CPU, 23450732Ki memory allocatable (of 12000m CPU, 24601708Ki memory capacity) 2026/07/16 15:54:59 Node: ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/16 15:54:59 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 15:54:59 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 15:54:59 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 15:54:59 Condition Ready: True - kubelet is posting ready status 2026/07/16 15:54:59 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/16 15:54:59 Node: ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/16 15:54:59 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 15:54:59 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 15:54:59 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 15:54:59 Condition Ready: True - kubelet is posting ready status 2026/07/16 15:54:59 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/16 15:54:59 Node: ods-qe-psi-07-b4mqp-worker-0-jd4pm 2026/07/16 15:54:59 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 15:54:59 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 15:54:59 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 15:54:59 Condition Ready: True - kubelet is posting ready status 2026/07/16 15:54:59 Resources: 11500m CPU, 23452736Ki memory allocatable (of 12000m CPU, 24603712Ki memory capacity) 2026/07/16 15:54:59 Node: ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 15:54:59 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 15:54:59 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 15:54:59 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 15:54:59 Condition Ready: True - kubelet is posting ready status 2026/07/16 15:54:59 Resources: 11500m CPU, 23452744Ki memory allocatable (of 12000m CPU, 24603720Ki memory capacity) 2026/07/16 15:54:59 Node: ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/16 15:54:59 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 15:54:59 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 15:54:59 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 15:54:59 Condition Ready: True - kubelet is posting ready status 2026/07/16 15:54:59 Resources: 11500m CPU, 23450720Ki memory allocatable (of 12000m CPU, 24601696Ki memory capacity) 2026/07/16 15:54:59 === DEPLOYMENTS === 2026/07/16 15:54:59 Namespace: redhat-ods-applications 2026/07/16 15:54:59 Deployment maas-api: 1/1 (READY) 2026/07/16 15:54:59 Deployment maas-controller: 1/1 (READY) 2026/07/16 15:54:59 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/16 15:54:59 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/16 15:54:59 Namespace: kube-system 2026/07/16 15:54:59 === PODS === 2026/07/16 15:54:59 Namespace kube-system: no problematic pods found 2026/07/16 15:54:59 Namespace redhat-ods-applications: no problematic pods found 2026/07/16 15:54:59 === OPERATOR STATUS === 2026/07/16 15:54:59 Operator deployment: 3/3 ready 2026/07/16 15:54:59 === DSCI STATUS === 2026/07/16 15:54:59 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/16 15:54:59 === DSC STATUS === 2026/07/16 15:54:59 Failed to collect DSC data: default-dsc unhealthy conditions: ModelsAsServiceReady=False 2026/07/16 15:54:59 === RECENT EVENTS === 2026/07/16 15:54:59 15:54:52 WARN Pod/odh-model-controller-867855946f-xss88: Unhealthy - Readiness probe failed: Get "http://10.130.2.28:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026/07/16 15:54:59 15:54:51 INFO Pod/odh-model-controller-867855946f-xss88: Killing - Stopping container manager 2026/07/16 15:54:59 15:54:51 INFO Pod/model-registry-operator-controller-manager-76b5b7f674-gxbds: Killing - Stopping container manager 2026/07/16 15:54:59 15:54:51 INFO Pod/model-serving-api-6688cd56bd-mp9jk: Killing - Stopping container server 2026/07/16 15:54:59 15:54:50 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-92d27: Killing - Stopping container manager 2026/07/16 15:54:59 15:54:50 INFO Pod/llmisvc-controller-manager-5d65df97bb-d5x2x: Killing - Stopping container manager 2026/07/16 15:54:59 15:54:49 WARN ServiceMonitor/workload-variant-autoscaler-controller-manager-metrics-monitor: InvalidConfiguration - ServiceMonitor workload-variant-autoscaler-controller-manager-metrics-monitor was rejected due to invalid configuration: unable to get secret "workload-variant-autoscaler-controller-manager-token": secrets "workload-variant-autoscaler-controller-manager-token" not found 2026/07/16 15:54:59 15:54:49 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - ServiceMonitor controller-manager-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 15:54:59 15:54:49 INFO Pod/kserve-controller-manager-6c67dcb69c-6lvk4: Killing - Stopping container manager 2026/07/16 15:54:59 15:54:36 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-867855946f-xss88_a07ad278-a518-4d0a-8784-44d8285ed22d became leader 2026/07/16 15:54:59 15:54:35 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-5bd688895b-92d27_d8812411-aff1-47e2-9d60-0214e720534f became leader 2026/07/16 15:54:59 15:54:34 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-92d27: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-5bd688895b-92d27 to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 15:54:59 15:54:34 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-92d27: Pulled - Container image "registry.redhat.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" already present on machine 2026/07/16 15:54:59 15:54:34 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-92d27: AddedInterface - Add eth0 [10.130.2.29/23] from ovn-kubernetes 2026/07/16 15:54:59 15:54:34 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-92d27: Started - Started container manager 2026/07/16 15:54:59 15:54:34 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-5bd688895b-92d27 2026/07/16 15:54:59 15:54:34 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-92d27: Created - Created container: manager 2026/07/16 15:54:59 15:54:34 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-5bd688895b from 0 to 1 2026/07/16 15:54:59 15:54:34 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-hf6hn: Killing - Stopping container manager 2026/07/16 15:54:59 15:54:34 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-vf5l2: Killing - Stopping container manager 2026/07/16 15:54:59 15:54:20 INFO Pod/odh-model-controller-867855946f-xss88: Created - Created container: manager 2026/07/16 15:54:59 15:54:20 INFO Pod/odh-model-controller-867855946f-xss88: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 851ms (851ms including waiting). Image size: 238959091 bytes. 2026/07/16 15:54:59 15:54:20 INFO Pod/odh-model-controller-867855946f-xss88: Started - Started container manager 2026/07/16 15:54:59 15:54:19 INFO Pod/odh-model-controller-867855946f-xss88: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/16 15:54:59 15:54:19 INFO Pod/odh-model-controller-867855946f-xss88: AddedInterface - Add eth0 [10.130.2.28/23] from ovn-kubernetes 2026/07/16 15:54:59 15:54:18 INFO Pod/odh-model-controller-867855946f-xss88: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-867855946f-xss88 to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 15:54:59 15:54:18 INFO Pod/odh-model-controller-867855946f-w69ds: Killing - Stopping container manager 2026/07/16 15:54:59 15:54:18 INFO ReplicaSet/odh-model-controller-867855946f: SuccessfulCreate - Created pod: odh-model-controller-867855946f-xss88 2026/07/16 15:54:59 15:54:18 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-867855946f from 0 to 1 2026/07/16 15:54:59 15:54:18 INFO Pod/odh-model-controller-867855946f-gm7tg: Killing - Stopping container manager 2026/07/16 15:54:59 15:54:04 INFO Pod/model-serving-api-6688cd56bd-mp9jk: AddedInterface - Add eth0 [10.130.2.27/23] from ovn-kubernetes 2026/07/16 15:54:59 15:54:04 INFO Pod/model-serving-api-6688cd56bd-mp9jk: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/16 15:54:59 15:54:04 INFO Pod/model-serving-api-6688cd56bd-mp9jk: Created - Created container: server 2026/07/16 15:54:59 15:54:04 INFO Pod/model-serving-api-6688cd56bd-mp9jk: Started - Started container server 2026/07/16 15:54:59 15:54:03 INFO Pod/model-serving-api-6688cd56bd-mp9jk: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-mp9jk to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 15:54:59 15:54:03 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/16 15:54:59 15:54:03 INFO Pod/model-serving-api-6688cd56bd-4wrwm: Killing - Stopping container server 2026/07/16 15:54:59 15:54:03 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-mp9jk 2026/07/16 15:54:59 15:54:03 INFO Pod/model-serving-api-6688cd56bd-2w2nj: Killing - Stopping container server 2026/07/16 15:54:59 15:53:03 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-vf5l2: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-5bd688895b-vf5l2 to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/16 15:54:59 15:53:03 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-5bd688895b-vf5l2 2026/07/16 15:54:59 15:53:03 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-vf5l2: Started - Started container manager 2026/07/16 15:54:59 15:53:03 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-5bd688895b from 1 to 2 2026/07/16 15:54:59 15:53:03 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-vf5l2: AddedInterface - Add eth0 [10.131.0.17/23] from ovn-kubernetes 2026/07/16 15:54:59 15:53:03 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-vf5l2: Pulled - Container image "registry.redhat.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" already present on machine 2026/07/16 15:54:59 15:53:03 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-vf5l2: Created - Created container: manager 2026/07/16 15:54:59 15:52:46 INFO Pod/odh-model-controller-867855946f-gm7tg: Started - Started container manager 2026/07/16 15:54:59 15:52:46 INFO Pod/odh-model-controller-867855946f-gm7tg: Created - Created container: manager 2026/07/16 15:54:59 15:52:46 INFO Pod/odh-model-controller-867855946f-gm7tg: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 2.393s (2.393s including waiting). Image size: 238959091 bytes. 2026/07/16 15:54:59 15:52:43 INFO Pod/odh-model-controller-867855946f-gm7tg: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-867855946f-gm7tg to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 15:54:59 15:52:43 INFO Pod/odh-model-controller-867855946f-gm7tg: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/16 15:54:59 15:52:43 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-867855946f from 1 to 2 2026/07/16 15:54:59 15:52:43 INFO Pod/odh-model-controller-867855946f-gm7tg: AddedInterface - Add eth0 [10.130.2.26/23] from ovn-kubernetes 2026/07/16 15:54:59 15:52:43 INFO ReplicaSet/odh-model-controller-867855946f: SuccessfulCreate - Created pod: odh-model-controller-867855946f-gm7tg 2026/07/16 15:54:59 15:52:23 INFO Pod/model-serving-api-6688cd56bd-2w2nj: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-2w2nj to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 15:54:59 15:52:23 INFO Pod/model-serving-api-6688cd56bd-2w2nj: AddedInterface - Add eth0 [10.130.2.25/23] from ovn-kubernetes 2026/07/16 15:54:59 15:52:23 INFO Pod/model-serving-api-6688cd56bd-2w2nj: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/16 15:54:59 15:52:23 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-2w2nj 2026/07/16 15:54:59 15:52:23 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 1 to 2 2026/07/16 15:54:59 15:52:23 INFO Pod/model-serving-api-6688cd56bd-2w2nj: Started - Started container server 2026/07/16 15:54:59 15:52:23 INFO Pod/model-serving-api-6688cd56bd-2w2nj: Created - Created container: server 2026/07/16 15:54:59 15:52:12 INFO Pod/maas-api-84449bdd5d-g7ch2: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" in 1.992s (1.992s including waiting). Image size: 175018396 bytes. 2026/07/16 15:54:59 15:52:12 INFO Pod/maas-api-84449bdd5d-g7ch2: Started - Started container maas-api 2026/07/16 15:54:59 15:52:12 INFO Pod/maas-api-84449bdd5d-g7ch2: Created - Created container: maas-api 2026/07/16 15:54:59 15:52:10 INFO Pod/maas-api-84449bdd5d-g7ch2: AddedInterface - Add eth0 [10.130.2.22/23] from ovn-kubernetes 2026/07/16 15:54:59 15:52:10 INFO Pod/maas-api-84449bdd5d-g7ch2: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" 2026/07/16 15:54:59 15:52:09 INFO Pod/maas-api-84449bdd5d-g7ch2: Scheduled - Successfully assigned redhat-ods-applications/maas-api-84449bdd5d-g7ch2 to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 15:54:59 15:52:09 INFO Deployment/maas-api: ScalingReplicaSet - Scaled up replica set maas-api-84449bdd5d from 0 to 1 2026/07/16 15:54:59 15:52:09 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 15:54:59 15:52:09 WARN Pod/maas-api-84449bdd5d-g7ch2: FailedMount - MountVolume.SetUp failed for volume "maas-api-tls" : secret "maas-api-serving-cert" not found 2026/07/16 15:54:59 15:52:09 WARN ReplicaSet/maas-api-84449bdd5d: FailedCreate - Error creating: pods "maas-api-84449bdd5d-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 2026/07/16 15:54:59 15:52:09 INFO ReplicaSet/maas-api-84449bdd5d: SuccessfulCreate - Created pod: maas-api-84449bdd5d-g7ch2 2026/07/16 15:54:59 15:51:56 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-d5x2x_f233f9c9-782f-4f93-bfc3-49bc5175e83f became leader 2026/07/16 15:54:59 15:51:55 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-6lvk4_05ca4dc3-daa2-4b9b-9a65-3d2ae2f8be70 became leader 2026/07/16 15:54:59 15:51:45 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-5bd688895b-hf6hn_0332a011-e83c-4cd6-a218-861809c65b1c became leader 2026/07/16 15:54:59 15:51:45 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-hf6hn: Created - Created container: manager 2026/07/16 15:54:59 15:51:45 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-hf6hn: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" in 2.02s (2.02s including waiting). Image size: 195240093 bytes. 2026/07/16 15:54:59 15:51:45 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-hf6hn: Started - Started container manager 2026/07/16 15:54:59 15:51:42 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-hf6hn: Pulling - Pulling image "registry.redhat.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" 2026/07/16 15:54:59 15:51:42 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-hf6hn: AddedInterface - Add eth0 [10.130.2.20/23] from ovn-kubernetes 2026/07/16 15:54:59 15:51:41 WARN Pod/workload-variant-autoscaler-controller-manager-5bd688895b-hf6hn: FailedMount - MountVolume.SetUp failed for volume "epp-metrics-token" : secret "workload-variant-autoscaler-epp-metrics-token" not found 2026/07/16 15:54:59 15:51:40 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-hf6hn: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-5bd688895b-hf6hn to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 15:54:59 15:51:40 WARN ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: FailedCreate - Error creating: pods "workload-variant-autoscaler-controller-manager-5bd688895b-" is forbidden: error looking up service account redhat-ods-applications/workload-variant-autoscaler-controller-manager: serviceaccount "workload-variant-autoscaler-controller-manager" not found 2026/07/16 15:54:59 15:51:40 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-5bd688895b from 0 to 1 2026/07/16 15:54:59 15:51:40 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-5bd688895b-hf6hn 2026/07/16 15:54:59 15:51:38 INFO Pod/kserve-controller-manager-6c67dcb69c-6lvk4: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 887ms (887ms including waiting). Image size: 238435613 bytes. 2026/07/16 15:54:59 15:51:38 INFO Pod/kserve-controller-manager-6c67dcb69c-6lvk4: Started - Started container manager 2026/07/16 15:54:59 15:51:38 INFO Pod/kserve-controller-manager-6c67dcb69c-6lvk4: Created - Created container: manager 2026/07/16 15:54:59 15:51:37 INFO Pod/kserve-controller-manager-6c67dcb69c-6lvk4: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/16 15:54:59 15:51:37 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-867855946f-w69ds_daf77a38-b83a-4a60-be56-6b1224261901 became leader 2026/07/16 15:54:59 15:51:37 INFO Pod/llmisvc-controller-manager-5d65df97bb-d5x2x: Started - Started container manager 2026/07/16 15:54:59 15:51:37 INFO Pod/kserve-controller-manager-6c67dcb69c-6lvk4: AddedInterface - Add eth0 [10.131.0.15/23] from ovn-kubernetes 2026/07/16 15:54:59 15:51:37 INFO Lease/maas-controller.models-as-a-service.opendatahub.io: LeaderElection - maas-controller-85bdcf9988-bgg26_029d1db7-61e1-4416-9e46-c0c156c86aec became leader 2026/07/16 15:54:59 15:51:37 INFO Pod/llmisvc-controller-manager-5d65df97bb-d5x2x: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 794ms (794ms including waiting). Image size: 236366599 bytes. 2026/07/16 15:54:59 15:51:37 INFO Pod/llmisvc-controller-manager-5d65df97bb-d5x2x: Created - Created container: manager 2026/07/16 15:54:59 15:51:36 INFO Pod/llmisvc-controller-manager-5d65df97bb-d5x2x: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-d5x2x to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/16 15:54:59 15:51:36 INFO Pod/kserve-controller-manager-6c67dcb69c-6lvk4: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-6lvk4 to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/16 15:54:59 15:51:36 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-d5x2x 2026/07/16 15:54:59 15:51:36 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-6lvk4 2026/07/16 15:54:59 15:51:36 INFO Lease/85f368d1.opendatahub.io: LeaderElection - model-registry-operator-controller-manager-76b5b7f674-gxbds_583e1fa6-af9e-45f7-ac1f-119a7c80e266 became leader 2026/07/16 15:54:59 15:51:36 WARN Pod/kserve-controller-manager-6c67dcb69c-6lvk4: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 2026/07/16 15:54:59 15:51:36 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/16 15:54:59 15:51:36 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/16 15:54:59 15:51:36 INFO Pod/llmisvc-controller-manager-5d65df97bb-d5x2x: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/16 15:54:59 15:51:36 INFO Pod/llmisvc-controller-manager-5d65df97bb-d5x2x: AddedInterface - Add eth0 [10.129.2.178/23] from ovn-kubernetes 2026/07/16 15:54:59 15:51:22 INFO Pod/model-serving-api-6688cd56bd-4wrwm: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-4wrwm to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/16 15:54:59 15:51:22 INFO Pod/model-serving-api-6688cd56bd-4wrwm: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/16 15:54:59 15:51:22 INFO Pod/odh-model-controller-867855946f-w69ds: Started - Started container manager 2026/07/16 15:54:59 15:51:22 INFO Pod/maas-controller-85bdcf9988-bgg26: Created - Created container: manager 2026/07/16 15:54:59 15:51:22 INFO Pod/maas-controller-85bdcf9988-bgg26: Started - Started container manager 2026/07/16 15:54:59 15:51:22 INFO Pod/odh-model-controller-867855946f-w69ds: Created - Created container: manager 2026/07/16 15:54:59 15:51:22 INFO Pod/odh-model-controller-867855946f-w69ds: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 758ms (758ms including waiting). Image size: 238959091 bytes. 2026/07/16 15:54:59 15:51:22 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/16 15:54:59 15:51:22 INFO Pod/model-serving-api-6688cd56bd-4wrwm: AddedInterface - Add eth0 [10.131.0.14/23] from ovn-kubernetes 2026/07/16 15:54:59 15:51:22 INFO Pod/model-serving-api-6688cd56bd-4wrwm: Created - Created container: server 2026/07/16 15:54:59 15:51:22 INFO Pod/model-serving-api-6688cd56bd-4wrwm: Started - Started container server 2026/07/16 15:54:59 15:51:22 INFO Pod/maas-controller-85bdcf9988-bgg26: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 1.919s (1.919s including waiting). Image size: 212866077 bytes. 2026/07/16 15:54:59 15:51:22 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-4wrwm 2026/07/16 15:54:59 15:51:22 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/16 15:54:59 15:51:21 INFO Pod/odh-model-controller-867855946f-w69ds: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/16 15:54:59 15:51:21 INFO Pod/odh-model-controller-867855946f-w69ds: AddedInterface - Add eth0 [10.129.2.177/23] from ovn-kubernetes 2026/07/16 15:54:59 15:51:20 INFO Pod/maas-controller-85bdcf9988-bgg26: Scheduled - Successfully assigned redhat-ods-applications/maas-controller-85bdcf9988-bgg26 to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 15:54:59 15:51:20 INFO Pod/model-registry-operator-controller-manager-76b5b7f674-gxbds: Pulled - Container image "registry.redhat.io/rhoai/odh-model-registry-operator-rhel9@sha256:ebcec5c43468283ed3cbab50530f939166f8f3e8c9d306e91a8107d05d7fa9d4" already present on machine 2026/07/16 15:54:59 15:51:20 INFO Pod/model-registry-operator-controller-manager-76b5b7f674-gxbds: Created - Created container: manager 2026/07/16 15:54:59 15:51:20 WARN Pod/odh-model-controller-867855946f-w69ds: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/16 15:54:59 15:51:20 INFO Pod/maas-controller-85bdcf9988-bgg26: AddedInterface - Add eth0 [10.130.2.18/23] from ovn-kubernetes 2026/07/16 15:54:59 15:51:20 INFO Pod/maas-controller-85bdcf9988-bgg26: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/16 15:54:59 15:51:20 INFO ReplicaSet/maas-controller-85bdcf9988: SuccessfulCreate - Created pod: maas-controller-85bdcf9988-bgg26 2026/07/16 15:54:59 15:51:20 INFO Deployment/maas-controller: ScalingReplicaSet - Scaled up replica set maas-controller-85bdcf9988 from 0 to 1 2026/07/16 15:54:59 15:51:20 INFO Pod/model-registry-operator-controller-manager-76b5b7f674-gxbds: Started - Started container manager 2026/07/16 15:54:59 15:51:20 INFO Pod/model-registry-operator-controller-manager-76b5b7f674-gxbds: AddedInterface - Add eth0 [10.129.2.176/23] from ovn-kubernetes 2026/07/16 15:54:59 15:51:19 INFO Pod/odh-model-controller-867855946f-w69ds: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-867855946f-w69ds to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/16 15:54:59 15:51:19 INFO Pod/model-registry-operator-controller-manager-76b5b7f674-gxbds: Scheduled - Successfully assigned redhat-ods-applications/model-registry-operator-controller-manager-76b5b7f674-gxbds to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/16 15:54:59 15:51:19 WARN Pod/model-registry-operator-controller-manager-76b5b7f674-gxbds: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 2026/07/16 15:54:59 15:51:19 INFO ReplicaSet/model-registry-operator-controller-manager-76b5b7f674: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-76b5b7f674-gxbds 2026/07/16 15:54:59 15:51:19 INFO Deployment/model-registry-operator-controller-manager: ScalingReplicaSet - Scaled up replica set model-registry-operator-controller-manager-76b5b7f674 from 0 to 1 2026/07/16 15:54:59 15:51:19 INFO ReplicaSet/odh-model-controller-867855946f: SuccessfulCreate - Created pod: odh-model-controller-867855946f-w69ds 2026/07/16 15:54:59 15:51:19 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-867855946f from 0 to 1 2026/07/16 15:54:59 15:51:09 INFO Pod/dashboard-redirect-5bd755c985-nvwk4: Killing - Stopping container nginx 2026/07/16 15:54:59 15:51:09 INFO Pod/dashboard-redirect-5bd755c985-gclgs: Killing - Stopping container nginx 2026/07/16 15:54:59 15:50:29 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" in 815ms (815ms including waiting). Image size: 185092464 bytes. 2026/07/16 15:54:59 15:50:29 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" in 1.938s (1.938s including waiting). Image size: 185092464 bytes. 2026/07/16 15:54:59 15:50:28 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: Started - Started container mlflow-ui 2026/07/16 15:54:59 15:50:28 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" 2026/07/16 15:54:59 15:50:28 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: Started - Started container mlflow-ui 2026/07/16 15:54:59 15:50:28 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: Created - Created container: mlflow-ui 2026/07/16 15:54:59 15:50:28 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" 2026/07/16 15:54:59 15:50:28 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: Created - Created container: mlflow-ui 2026/07/16 15:54:59 15:50:28 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" in 800ms (800ms including waiting). Image size: 177808591 bytes. 2026/07/16 15:54:59 15:50:27 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" in 987ms (987ms including waiting). Image size: 187802205 bytes. 2026/07/16 15:54:59 15:50:27 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: Created - Created container: maas-ui 2026/07/16 15:54:59 15:50:27 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: Started - Started container maas-ui 2026/07/16 15:54:59 15:50:27 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" 2026/07/16 15:54:59 15:50:27 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" in 975ms (975ms including waiting). Image size: 177808591 bytes. 2026/07/16 15:54:59 15:50:26 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" in 752ms (752ms including waiting). Image size: 241242949 bytes. 2026/07/16 15:54:59 15:50:26 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" in 899ms (899ms including waiting). Image size: 187802205 bytes. 2026/07/16 15:54:59 15:50:26 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" 2026/07/16 15:54:59 15:50:26 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: Started - Started container gen-ai-ui 2026/07/16 15:54:59 15:50:26 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: Created - Created container: gen-ai-ui 2026/07/16 15:54:59 15:50:26 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" 2026/07/16 15:54:59 15:50:26 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: Started - Started container maas-ui 2026/07/16 15:54:59 15:50:26 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: Created - Created container: maas-ui 2026/07/16 15:54:59 15:50:25 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: Started - Started container model-registry-ui 2026/07/16 15:54:59 15:50:25 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" in 821ms (821ms including waiting). Image size: 182438697 bytes. 2026/07/16 15:54:59 15:50:25 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: Started - Started container gen-ai-ui 2026/07/16 15:54:59 15:50:25 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: Created - Created container: gen-ai-ui 2026/07/16 15:54:59 15:50:25 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" in 717ms (717ms including waiting). Image size: 241242949 bytes. 2026/07/16 15:54:59 15:50:25 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" 2026/07/16 15:54:59 15:50:25 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: Created - Created container: model-registry-ui 2026/07/16 15:54:59 15:50:25 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" 2026/07/16 15:54:59 15:50:25 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" in 733ms (733ms including waiting). Image size: 182438697 bytes. 2026/07/16 15:54:59 15:50:25 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: Started - Started container model-registry-ui 2026/07/16 15:54:59 15:50:25 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: Created - Created container: model-registry-ui 2026/07/16 15:54:59 15:50:25 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" 2026/07/16 15:54:59 15:50:24 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: Pulled - Container image "registry.redhat.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:1e614e46799fd020ac18d95c06a463421154ce86e59ccb3fa6007e66fed9910e" already present on machine 2026/07/16 15:54:59 15:50:24 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" 2026/07/16 15:54:59 15:50:24 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: Pulled - Container image "registry.redhat.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:1e614e46799fd020ac18d95c06a463421154ce86e59ccb3fa6007e66fed9910e" already present on machine 2026/07/16 15:54:59 15:50:24 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: Created - Created container: kube-rbac-proxy 2026/07/16 15:54:59 15:50:24 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: Created - Created container: kube-rbac-proxy 2026/07/16 15:54:59 15:50:24 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: Started - Started container kube-rbac-proxy 2026/07/16 15:54:59 15:50:24 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: Started - Started container kube-rbac-proxy 2026/07/16 15:54:59 15:50:24 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" in 1.407s (1.407s including waiting). Image size: 799822033 bytes. 2026/07/16 15:54:59 15:50:24 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: Started - Started container rhods-dashboard 2026/07/16 15:54:59 15:50:24 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: Created - Created container: rhods-dashboard 2026/07/16 15:54:59 15:50:24 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" in 890ms (890ms including waiting). Image size: 799822033 bytes. 2026/07/16 15:54:59 15:50:24 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" 2026/07/16 15:54:59 15:50:24 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: Started - Started container rhods-dashboard 2026/07/16 15:54:59 15:50:24 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: Created - Created container: rhods-dashboard 2026/07/16 15:54:59 15:50:23 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: AddedInterface - Add eth0 [10.130.2.17/23] from ovn-kubernetes 2026/07/16 15:54:59 15:50:23 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: Pulling - Pulling image "registry.redhat.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" 2026/07/16 15:54:59 15:50:23 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: Pulling - Pulling image "registry.redhat.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" 2026/07/16 15:54:59 15:50:23 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: AddedInterface - Add eth0 [10.129.2.175/23] from ovn-kubernetes 2026/07/16 15:54:59 15:50:22 INFO Pod/rhods-dashboard-d8f68f65-gpxr6: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-d8f68f65-gpxr6 to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 15:54:59 15:50:22 INFO Pod/rhods-dashboard-d8f68f65-zlbxq: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-d8f68f65-zlbxq to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/16 15:54:59 15:50:22 INFO Deployment/rhods-dashboard: ScalingReplicaSet - Scaled up replica set rhods-dashboard-d8f68f65 from 0 to 2 2026/07/16 15:54:59 15:50:22 INFO ReplicaSet/rhods-dashboard-d8f68f65: SuccessfulCreate - Created pod: rhods-dashboard-d8f68f65-gpxr6 2026/07/16 15:54:59 15:50:22 INFO ReplicaSet/rhods-dashboard-d8f68f65: SuccessfulCreate - Created pod: rhods-dashboard-d8f68f65-zlbxq 2026/07/16 15:54:59 15:50:20 INFO Pod/spark-operator-webhook-9cb9bdf44-qjbcx: Killing - Stopping container webhook 2026/07/16 15:54:59 15:50:20 INFO Pod/spark-operator-controller-767899d44f-fs5kq: Killing - Stopping container controller 2026/07/16 15:54:59 15:50:08 INFO Pod/maas-controller-85bdcf9988-b8hhc: Killing - Stopping container manager 2026/07/16 15:54:59 15:50:08 INFO Pod/maas-api-84449bdd5d-v5grq: Killing - Stopping container maas-api 2026/07/16 15:54:59 15:50:07 INFO Pod/odh-model-controller-6849fc49d8-24q4r: Killing - Stopping container manager 2026/07/16 15:54:59 15:50:07 INFO Pod/model-serving-api-6688cd56bd-8zmqk: Killing - Stopping container server 2026/07/16 15:54:59 15:50:07 INFO Pod/kserve-controller-manager-6c67dcb69c-8qt89: Killing - Stopping container manager 2026/07/16 15:54:59 15:50:07 INFO Pod/llmisvc-controller-manager-5d65df97bb-8zt4z: Killing - Stopping container manager 2026/07/16 15:54:59 15:50:06 INFO Pod/spark-operator-webhook-9cb9bdf44-qjbcx: Pulled - Container image "registry.redhat.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/16 15:54:59 15:50:06 INFO Pod/spark-operator-webhook-9cb9bdf44-qjbcx: Started - Started container webhook 2026/07/16 15:54:59 15:50:06 INFO Pod/spark-operator-webhook-9cb9bdf44-qjbcx: Created - Created container: webhook 2026/07/16 15:54:59 15:50:06 INFO Pod/spark-operator-webhook-9cb9bdf44-qjbcx: AddedInterface - Add eth0 [10.130.2.16/23] from ovn-kubernetes 2026/07/16 15:54:59 15:50:06 INFO Lease/spark-operator-controller-lock: LeaderElection - spark-operator-controller-767899d44f-fs5kq_7490ea40-814b-4232-baea-3a167d9c71a0 became leader 2026/07/16 15:54:59 15:50:05 INFO Pod/spark-operator-webhook-9cb9bdf44-qjbcx: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-webhook-9cb9bdf44-qjbcx to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 15:54:59 15:50:05 INFO Pod/spark-operator-webhook-9cb9bdf44-w7cv4: Killing - Stopping container webhook 2026/07/16 15:54:59 15:50:05 INFO ReplicaSet/spark-operator-webhook-9cb9bdf44: SuccessfulCreate - Created pod: spark-operator-webhook-9cb9bdf44-qjbcx 2026/07/16 15:54:59 15:50:05 INFO Deployment/spark-operator-webhook: ScalingReplicaSet - Scaled up replica set spark-operator-webhook-9cb9bdf44 from 0 to 1 2026/07/16 15:54:59 15:50:05 INFO Pod/spark-operator-webhook-9cb9bdf44-pd55d: Killing - Stopping container webhook 2026/07/16 15:54:59 === RESOURCE QUOTAS === 2026/07/16 15:54:59 No resource quotas found in monitored namespaces 2026/07/16 15:54:59 Classification for group 2: infrastructure/storage (code=1006, confidence=medium) 2026/07/16 15:54:59 Evidence: event Pod/maas-api-84449bdd5d-g7ch2: FailedMount - MountVolume.SetUp failed for volume "maas-api-tls" : secret "maas-api-serving-cert" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/maas-api-84449bdd5d-g7ch2: FailedMount - MountVolume.SetUp failed for volume \"maas-api-tls\" : secret \"maas-api-serving-cert\" not found"],"confidence":"medium"} 2026/07/16 15:54:59 Diagnostics complete for failed test: group 2 controller_test.go:753: Stopping: group 2 test failed. 2026/07/16 15:54:59 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/16 15:54:59 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/16 15:59:17 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (1033.03s) === RUN TestOdhOperator 2026/07/16 15:45:57 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/16 16:03:21 TEST FAILURE DETECTED: components 2026/07/16 16:03:21 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/16 16:03:21 === CLUSTER STATE === 2026/07/16 16:03:21 Node: ods-qe-psi-07-b4mqp-master-0 2026/07/16 16:03:21 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 16:03:21 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 16:03:21 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 16:03:21 Condition Ready: True - kubelet is posting ready status 2026/07/16 16:03:21 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/16 16:03:21 Node: ods-qe-psi-07-b4mqp-master-1 2026/07/16 16:03:21 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 16:03:21 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 16:03:21 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 16:03:21 Condition Ready: True - kubelet is posting ready status 2026/07/16 16:03:21 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/16 16:03:21 Node: ods-qe-psi-07-b4mqp-master-2 2026/07/16 16:03:21 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 16:03:21 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 16:03:21 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 16:03:21 Condition Ready: True - kubelet is posting ready status 2026/07/16 16:03:21 Resources: 11500m CPU, 23450732Ki memory allocatable (of 12000m CPU, 24601708Ki memory capacity) 2026/07/16 16:03:21 Node: ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/16 16:03:21 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 16:03:21 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 16:03:21 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 16:03:21 Condition Ready: True - kubelet is posting ready status 2026/07/16 16:03:21 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/16 16:03:21 Node: ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/16 16:03:21 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 16:03:21 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 16:03:21 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 16:03:21 Condition Ready: True - kubelet is posting ready status 2026/07/16 16:03:21 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/16 16:03:21 Node: ods-qe-psi-07-b4mqp-worker-0-jd4pm 2026/07/16 16:03:21 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 16:03:21 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 16:03:21 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 16:03:21 Condition Ready: True - kubelet is posting ready status 2026/07/16 16:03:21 Resources: 11500m CPU, 23452736Ki memory allocatable (of 12000m CPU, 24603712Ki memory capacity) 2026/07/16 16:03:21 Node: ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 16:03:21 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 16:03:21 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 16:03:21 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 16:03:21 Condition Ready: True - kubelet is posting ready status 2026/07/16 16:03:21 Resources: 11500m CPU, 23452744Ki memory allocatable (of 12000m CPU, 24603720Ki memory capacity) 2026/07/16 16:03:21 Node: ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/16 16:03:21 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 16:03:21 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 16:03:21 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 16:03:21 Condition Ready: True - kubelet is posting ready status 2026/07/16 16:03:21 Resources: 11500m CPU, 23450720Ki memory allocatable (of 12000m CPU, 24601696Ki memory capacity) 2026/07/16 16:03:21 === DEPLOYMENTS === 2026/07/16 16:03:21 Namespace: redhat-ods-applications 2026/07/16 16:03:21 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/16 16:03:21 Deployment kubeflow-trainer-controller-manager: 1/1 (READY) 2026/07/16 16:03:21 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/16 16:03:21 Deployment maas-postgres: 1/1 (READY) 2026/07/16 16:03:21 Deployment model-serving-api: 1/1 (READY) 2026/07/16 16:03:21 Deployment odh-model-controller: 1/1 (READY) 2026/07/16 16:03:21 Namespace: kube-system 2026/07/16 16:03:21 === PODS === 2026/07/16 16:03:21 Namespace redhat-ods-applications: no problematic pods found 2026/07/16 16:03:21 Namespace kube-system: no problematic pods found 2026/07/16 16:03:21 === OPERATOR STATUS === 2026/07/16 16:03:21 Operator deployment: 3/3 ready 2026/07/16 16:03:21 === DSCI STATUS === 2026/07/16 16:03:21 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/16 16:03:21 === DSC STATUS === 2026/07/16 16:03:21 DSC default-dsc: 2026/07/16 16:03:21 Ready: True - 2026/07/16 16:03:21 ProvisioningSucceeded: True - 2026/07/16 16:03:21 ComponentsReady: True - 2026/07/16 16:03:21 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/16 16:03:21 DashboardReady: False - Component ManagementState is set to Removed 2026/07/16 16:03:21 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/16 16:03:21 KserveLLMInferenceServiceDependencies: True - 2026/07/16 16:03:21 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/16 16:03:21 KserveReady: True - 2026/07/16 16:03:21 KueueReady: True - 2026/07/16 16:03:21 LLM-D-WVADependencies: True - 2026/07/16 16:03:21 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/16 16:03:21 ModelControllerReady: True - 2026/07/16 16:03:21 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/16 16:03:21 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/16 16:03:21 OGXReady: False - Component ManagementState is set to Removed 2026/07/16 16:03:21 RayReady: False - Component ManagementState is set to Removed 2026/07/16 16:03:21 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/16 16:03:21 TrainerReady: True - 2026/07/16 16:03:21 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/16 16:03:21 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/16 16:03:21 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/16 16:03:21 === RECENT EVENTS === 2026/07/16 16:03:21 16:02:57 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-4kjsl_852ff104-588e-4317-b242-f6487accad83 became leader 2026/07/16 16:03:21 16:02:53 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-78pvw_2dce4490-c7ed-4fa3-b089-54f87364b675 became leader 2026/07/16 16:03:21 16:02:39 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-p6cs8_ca72dd9e-82c9-4bd6-9c93-441ff9366f44 became leader 2026/07/16 16:03:21 16:02:38 INFO Pod/llmisvc-controller-manager-5d65df97bb-4kjsl: Started - Started container manager 2026/07/16 16:03:21 16:02:38 INFO Pod/llmisvc-controller-manager-5d65df97bb-4kjsl: Created - Created container: manager 2026/07/16 16:03:21 16:02:38 INFO Pod/llmisvc-controller-manager-5d65df97bb-4kjsl: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 2.378s (2.378s including waiting). Image size: 236366599 bytes. 2026/07/16 16:03:21 16:02:37 INFO Pod/kserve-controller-manager-6c67dcb69c-78pvw: Started - Started container manager 2026/07/16 16:03:21 16:02:37 INFO Pod/kserve-controller-manager-6c67dcb69c-78pvw: Created - Created container: manager 2026/07/16 16:03:21 16:02:36 INFO Pod/llmisvc-controller-manager-5d65df97bb-4kjsl: AddedInterface - Add eth0 [10.129.2.185/23] from ovn-kubernetes 2026/07/16 16:03:21 16:02:36 INFO Pod/kserve-controller-manager-6c67dcb69c-78pvw: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 780ms (780ms including waiting). Image size: 238435613 bytes. 2026/07/16 16:03:21 16:02:36 INFO Pod/kserve-controller-manager-6c67dcb69c-78pvw: AddedInterface - Add eth0 [10.130.2.54/23] from ovn-kubernetes 2026/07/16 16:03:21 16:02:36 INFO Pod/llmisvc-controller-manager-5d65df97bb-4kjsl: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/16 16:03:21 16:02:36 INFO Pod/kserve-controller-manager-6c67dcb69c-78pvw: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/16 16:03:21 16:02:35 INFO Pod/llmisvc-controller-manager-5d65df97bb-4kjsl: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-4kjsl to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/16 16:03:21 16:02:35 INFO Pod/kserve-controller-manager-6c67dcb69c-78pvw: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-78pvw to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 16:03:21 16:02:35 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/16 16:03:21 16:02:35 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/16 16:03:21 16:02:35 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-4kjsl 2026/07/16 16:03:21 16:02:35 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-78pvw 2026/07/16 16:03:21 16:02:23 INFO Pod/model-serving-api-6688cd56bd-8hxcb: Started - Started container server 2026/07/16 16:03:21 16:02:23 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 16:03:21 16:02:23 INFO Pod/model-serving-api-6688cd56bd-8hxcb: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/16 16:03:21 16:02:23 INFO Pod/model-serving-api-6688cd56bd-8hxcb: Created - Created container: server 2026/07/16 16:03:21 16:02:23 INFO Pod/model-serving-api-6688cd56bd-8hxcb: AddedInterface - Add eth0 [10.130.2.53/23] from ovn-kubernetes 2026/07/16 16:03:21 16:02:22 INFO Pod/model-serving-api-6688cd56bd-8hxcb: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-8hxcb to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 16:03:21 16:02:22 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/16 16:03:21 16:02:22 INFO Pod/odh-model-controller-6849fc49d8-p6cs8: Started - Started container manager 2026/07/16 16:03:21 16:02:22 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/16 16:03:21 16:02:22 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-8hxcb 2026/07/16 16:03:21 16:02:22 INFO Pod/odh-model-controller-6849fc49d8-p6cs8: Created - Created container: manager 2026/07/16 16:03:21 16:02:22 INFO Pod/odh-model-controller-6849fc49d8-p6cs8: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 863ms (863ms including waiting). Image size: 238959091 bytes. 2026/07/16 16:03:21 16:02:21 INFO Pod/odh-model-controller-6849fc49d8-p6cs8: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-p6cs8 to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 16:03:21 16:02:21 INFO Pod/odh-model-controller-6849fc49d8-p6cs8: AddedInterface - Add eth0 [10.130.2.52/23] from ovn-kubernetes 2026/07/16 16:03:21 16:02:21 INFO Pod/odh-model-controller-6849fc49d8-p6cs8: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/16 16:03:21 16:02:21 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-p6cs8 2026/07/16 16:03:21 16:02:20 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/16 16:03:21 16:02:11 WARN Endpoints/odh-model-controller-metrics-service: FailedToUpdateEndpoint - Failed to update endpoint redhat-ods-applications/odh-model-controller-metrics-service: Operation cannot be fulfilled on endpoints "odh-model-controller-metrics-service": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/odh-model-controller-metrics-service, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 461bd3cc-9d35-4b5c-85e3-3433cf8e6718, UID in object meta: 2026/07/16 16:03:21 16:02:11 INFO Pod/llmisvc-controller-manager-5d65df97bb-6vx79: Killing - Stopping container manager 2026/07/16 16:03:21 16:02:11 INFO Pod/odh-model-controller-6849fc49d8-wswvx: Killing - Stopping container manager 2026/07/16 16:03:21 16:02:11 INFO Pod/model-serving-api-6688cd56bd-d2xqr: Killing - Stopping container server 2026/07/16 16:03:21 16:02:10 INFO Pod/kserve-controller-manager-6c67dcb69c-zrc22: Killing - Stopping container manager 2026/07/16 16:03:21 16:02:10 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 16:03:21 16:01:53 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-6vx79_1d895dd2-e969-482f-b54e-203468704900 became leader 2026/07/16 16:03:21 16:01:51 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-zrc22_4c5b8daf-cb4f-4fa3-9394-1d77a6f1ef3e became leader 2026/07/16 16:03:21 16:01:42 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-wswvx_946530ca-6a6a-4896-acd3-a26908534587 became leader 2026/07/16 16:03:21 16:01:40 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-7ccfdd66-hwgqz_16c2e9d4-08aa-4f13-b6bd-d1fb7956f64a became leader 2026/07/16 16:03:21 16:01:36 INFO Pod/llmisvc-controller-manager-5d65df97bb-6vx79: Started - Started container manager 2026/07/16 16:03:21 16:01:36 INFO Pod/llmisvc-controller-manager-5d65df97bb-6vx79: Created - Created container: manager 2026/07/16 16:03:21 16:01:36 INFO Pod/llmisvc-controller-manager-5d65df97bb-6vx79: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 2.164s (2.164s including waiting). Image size: 236366599 bytes. 2026/07/16 16:03:21 16:01:35 INFO Pod/kserve-controller-manager-6c67dcb69c-zrc22: Created - Created container: manager 2026/07/16 16:03:21 16:01:35 INFO Pod/kserve-controller-manager-6c67dcb69c-zrc22: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 979ms (979ms including waiting). Image size: 238435613 bytes. 2026/07/16 16:03:21 16:01:35 INFO Pod/kserve-controller-manager-6c67dcb69c-zrc22: Started - Started container manager 2026/07/16 16:03:21 16:01:34 INFO Pod/kserve-controller-manager-6c67dcb69c-zrc22: AddedInterface - Add eth0 [10.130.2.50/23] from ovn-kubernetes 2026/07/16 16:03:21 16:01:34 INFO Pod/kserve-controller-manager-6c67dcb69c-zrc22: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/16 16:03:21 16:01:34 INFO Pod/llmisvc-controller-manager-5d65df97bb-6vx79: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/16 16:03:21 16:01:34 INFO Pod/llmisvc-controller-manager-5d65df97bb-6vx79: AddedInterface - Add eth0 [10.130.2.51/23] from ovn-kubernetes 2026/07/16 16:03:21 16:01:33 INFO Pod/llmisvc-controller-manager-5d65df97bb-6vx79: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-6vx79 to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 16:03:21 16:01:33 INFO Pod/kserve-controller-manager-6c67dcb69c-zrc22: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-zrc22 to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 16:03:21 16:01:33 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-6vx79 2026/07/16 16:03:21 16:01:33 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/16 16:03:21 16:01:33 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-zrc22 2026/07/16 16:03:21 16:01:33 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/16 16:03:21 16:01:23 INFO Pod/model-serving-api-6688cd56bd-d2xqr: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-d2xqr to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 16:03:21 16:01:23 INFO Pod/model-serving-api-6688cd56bd-d2xqr: AddedInterface - Add eth0 [10.130.2.49/23] from ovn-kubernetes 2026/07/16 16:03:21 16:01:23 INFO Pod/model-serving-api-6688cd56bd-d2xqr: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/16 16:03:21 16:01:23 INFO Pod/odh-model-controller-6849fc49d8-wswvx: Started - Started container manager 2026/07/16 16:03:21 16:01:23 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-d2xqr 2026/07/16 16:03:21 16:01:23 INFO Pod/odh-model-controller-6849fc49d8-wswvx: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 1.276s (1.276s including waiting). Image size: 238959091 bytes. 2026/07/16 16:03:21 16:01:23 INFO Pod/model-serving-api-6688cd56bd-d2xqr: Started - Started container server 2026/07/16 16:03:21 16:01:23 INFO Pod/odh-model-controller-6849fc49d8-wswvx: Created - Created container: manager 2026/07/16 16:03:21 16:01:23 INFO Pod/model-serving-api-6688cd56bd-d2xqr: Created - Created container: server 2026/07/16 16:03:21 16:01:22 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/16 16:03:21 16:01:22 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/16 16:03:21 16:01:22 INFO Pod/odh-model-controller-6849fc49d8-wswvx: AddedInterface - Add eth0 [10.130.2.48/23] from ovn-kubernetes 2026/07/16 16:03:21 16:01:22 INFO Pod/odh-model-controller-6849fc49d8-wswvx: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/16 16:03:21 16:01:21 WARN Pod/odh-model-controller-6849fc49d8-wswvx: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/16 16:03:21 16:01:21 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-hwgqz: Started - Started container manager 2026/07/16 16:03:21 16:01:21 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-hwgqz: Created - Created container: manager 2026/07/16 16:03:21 16:01:21 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-hwgqz: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-trainer-rhel9@sha256:59ea523e2df9c1e705c755308ac11420682081cc63d54890e2b15fbd7d8ba570" in 6.018s (6.018s including waiting). Image size: 157432248 bytes. 2026/07/16 16:03:21 16:01:20 INFO Pod/odh-model-controller-6849fc49d8-wswvx: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-wswvx to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 16:03:21 16:01:20 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-wswvx 2026/07/16 16:03:21 16:01:20 INFO Pod/llmisvc-controller-manager-5d65df97bb-wdbz4: Killing - Stopping container manager 2026/07/16 16:03:21 16:01:20 INFO Pod/llmisvc-controller-manager-5d65df97bb-wdbz4: Started - Started container manager 2026/07/16 16:03:21 16:01:20 INFO Pod/llmisvc-controller-manager-5d65df97bb-wdbz4: Created - Created container: manager 2026/07/16 16:03:21 16:01:20 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/16 16:03:21 16:01:20 INFO Pod/llmisvc-controller-manager-5d65df97bb-wdbz4: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 1.554s (1.554s including waiting). Image size: 236366599 bytes. 2026/07/16 16:03:21 16:01:19 INFO Pod/kserve-controller-manager-6c67dcb69c-49kdc: Created - Created container: manager 2026/07/16 16:03:21 16:01:19 INFO Pod/kserve-controller-manager-6c67dcb69c-49kdc: Started - Started container manager 2026/07/16 16:03:21 16:01:19 INFO Pod/kserve-controller-manager-6c67dcb69c-49kdc: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 1.031s (1.031s including waiting). Image size: 238435613 bytes. 2026/07/16 16:03:21 16:01:18 INFO Pod/llmisvc-controller-manager-5d65df97bb-wdbz4: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/16 16:03:21 16:01:18 INFO Pod/llmisvc-controller-manager-5d65df97bb-wdbz4: AddedInterface - Add eth0 [10.130.2.47/23] from ovn-kubernetes 2026/07/16 16:03:21 16:01:18 INFO Pod/kserve-controller-manager-6c67dcb69c-49kdc: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/16 16:03:21 16:01:18 INFO Pod/kserve-controller-manager-6c67dcb69c-49kdc: AddedInterface - Add eth0 [10.130.2.46/23] from ovn-kubernetes 2026/07/16 16:03:21 16:01:17 INFO Pod/llmisvc-controller-manager-5d65df97bb-wdbz4: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-wdbz4 to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 16:03:21 16:01:17 INFO Pod/kserve-controller-manager-6c67dcb69c-49kdc: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-49kdc to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 16:03:21 16:01:17 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/16 16:03:21 16:01:17 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-wdbz4 2026/07/16 16:03:21 16:01:17 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-49kdc 2026/07/16 16:03:21 16:01:17 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/16 16:03:21 16:01:15 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-hwgqz: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-7ccfdd66-hwgqz to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 16:03:21 16:01:15 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-7ccfdd66 from 0 to 1 2026/07/16 16:03:21 16:01:15 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-hwgqz: AddedInterface - Add eth0 [10.130.2.45/23] from ovn-kubernetes 2026/07/16 16:03:21 16:01:15 INFO ReplicaSet/kubeflow-trainer-controller-manager-7ccfdd66: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-7ccfdd66-hwgqz 2026/07/16 16:03:21 16:01:15 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-hwgqz: Pulling - Pulling image "registry.redhat.io/rhoai/odh-trainer-rhel9@sha256:59ea523e2df9c1e705c755308ac11420682081cc63d54890e2b15fbd7d8ba570" 2026/07/16 16:03:21 16:01:13 INFO Pod/kserve-controller-manager-6c67dcb69c-x8dxl: Killing - Stopping container manager 2026/07/16 16:03:21 16:01:13 WARN Pod/odh-model-controller-6849fc49d8-nzgnp: Unhealthy - Readiness probe failed: Get "http://10.130.2.39:8081/readyz": dial tcp 10.130.2.39:8081: connect: connection refused 2026/07/16 16:03:21 16:01:13 INFO Pod/odh-model-controller-6849fc49d8-nzgnp: Killing - Stopping container manager 2026/07/16 16:03:21 16:01:13 WARN Pod/model-serving-api-6688cd56bd-jr8hz: Unhealthy - Readiness probe failed: Get "https://10.130.2.41:8443/readyz": dial tcp 10.130.2.41:8443: connect: connection refused 2026/07/16 16:03:21 16:01:13 INFO Pod/model-serving-api-6688cd56bd-jr8hz: Killing - Stopping container server 2026/07/16 16:03:21 16:01:13 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 16:03:21 16:01:13 INFO Pod/llmisvc-controller-manager-5d65df97bb-nsbgg: Killing - Stopping container manager 2026/07/16 16:03:21 16:01:02 INFO Pod/maas-api-84449bdd5d-x259p: Killing - Stopping container maas-api 2026/07/16 16:03:21 16:01:01 INFO Pod/maas-controller-85bdcf9988-7p5hx: Killing - Stopping container manager 2026/07/16 16:03:21 16:00:31 INFO Pod/maas-api-84449bdd5d-x259p: Started - Started container maas-api 2026/07/16 16:03:21 16:00:31 INFO Pod/maas-api-84449bdd5d-x259p: Created - Created container: maas-api 2026/07/16 16:03:21 16:00:30 INFO Pod/maas-api-84449bdd5d-x259p: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" in 986ms (986ms including waiting). Image size: 175018396 bytes. 2026/07/16 16:03:21 16:00:29 INFO Pod/maas-api-84449bdd5d-x259p: Scheduled - Successfully assigned redhat-ods-applications/maas-api-84449bdd5d-x259p to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 16:03:21 16:00:29 WARN ReplicaSet/maas-api-84449bdd5d: FailedCreate - Error creating: pods "maas-api-84449bdd5d-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 2026/07/16 16:03:21 16:00:29 INFO Pod/maas-api-84449bdd5d-x259p: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" 2026/07/16 16:03:21 16:00:29 INFO Deployment/maas-api: ScalingReplicaSet - Scaled up replica set maas-api-84449bdd5d from 0 to 1 2026/07/16 16:03:21 16:00:29 INFO ReplicaSet/maas-api-84449bdd5d: SuccessfulCreate - Created pod: maas-api-84449bdd5d-x259p 2026/07/16 16:03:21 16:00:29 INFO Pod/maas-api-84449bdd5d-x259p: AddedInterface - Add eth0 [10.130.2.44/23] from ovn-kubernetes 2026/07/16 16:03:21 16:00:20 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-nsbgg_76f4c0ff-188d-4308-8754-dc45de67a24b became leader 2026/07/16 16:03:21 16:00:15 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-x8dxl_e2e18050-ff7c-4725-ac4d-a5667b49ee46 became leader 2026/07/16 16:03:21 16:00:00 INFO Pod/llmisvc-controller-manager-5d65df97bb-nsbgg: Created - Created container: manager 2026/07/16 16:03:21 16:00:00 INFO Pod/llmisvc-controller-manager-5d65df97bb-nsbgg: Started - Started container manager 2026/07/16 16:03:21 16:00:00 INFO Pod/llmisvc-controller-manager-5d65df97bb-nsbgg: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 3.315s (3.315s including waiting). Image size: 236366599 bytes. 2026/07/16 16:03:21 15:59:58 INFO Pod/kserve-controller-manager-6c67dcb69c-x8dxl: Created - Created container: manager 2026/07/16 16:03:21 15:59:58 INFO Pod/kserve-controller-manager-6c67dcb69c-x8dxl: Started - Started container manager 2026/07/16 16:03:21 15:59:58 INFO Pod/kserve-controller-manager-6c67dcb69c-x8dxl: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 1.204s (1.204s including waiting). Image size: 238435613 bytes. 2026/07/16 16:03:21 15:59:58 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-nzgnp_4cb6be79-0b4a-4a36-87ff-347b35c46d3c became leader 2026/07/16 16:03:21 15:59:57 INFO Pod/llmisvc-controller-manager-5d65df97bb-nsbgg: AddedInterface - Add eth0 [10.130.2.43/23] from ovn-kubernetes 2026/07/16 16:03:21 15:59:57 INFO Pod/llmisvc-controller-manager-5d65df97bb-nsbgg: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/16 16:03:21 15:59:57 INFO Lease/maas-controller.models-as-a-service.opendatahub.io: LeaderElection - maas-controller-85bdcf9988-7p5hx_0cde5cc2-e80f-443d-90a1-5b9dae846771 became leader 2026/07/16 16:03:21 15:59:57 INFO Pod/kserve-controller-manager-6c67dcb69c-x8dxl: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/16 16:03:21 15:59:57 INFO Pod/kserve-controller-manager-6c67dcb69c-x8dxl: AddedInterface - Add eth0 [10.130.2.42/23] from ovn-kubernetes 2026/07/16 16:03:21 15:59:56 WARN Pod/llmisvc-controller-manager-5d65df97bb-nsbgg: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 2026/07/16 16:03:21 15:59:56 WARN Pod/kserve-controller-manager-6c67dcb69c-x8dxl: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 2026/07/16 16:03:21 15:59:55 INFO Pod/llmisvc-controller-manager-5d65df97bb-nsbgg: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-nsbgg to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 16:03:21 15:59:55 INFO Pod/kserve-controller-manager-6c67dcb69c-x8dxl: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-x8dxl to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 16:03:21 15:59:55 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-x8dxl 2026/07/16 16:03:21 15:59:55 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-nsbgg 2026/07/16 16:03:21 15:59:55 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/16 16:03:21 15:59:55 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/16 16:03:21 15:59:41 INFO Pod/model-serving-api-6688cd56bd-jr8hz: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-jr8hz to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 16:03:21 15:59:41 INFO Pod/maas-controller-85bdcf9988-7p5hx: Created - Created container: manager 2026/07/16 16:03:21 15:59:41 INFO Pod/maas-controller-85bdcf9988-7p5hx: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 857ms (857ms including waiting). Image size: 212866077 bytes. 2026/07/16 16:03:21 15:59:41 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/16 16:03:21 15:59:41 INFO Pod/maas-controller-85bdcf9988-7p5hx: Started - Started container manager 2026/07/16 16:03:21 15:59:41 INFO Pod/model-serving-api-6688cd56bd-jr8hz: AddedInterface - Add eth0 [10.130.2.41/23] from ovn-kubernetes 2026/07/16 16:03:21 15:59:41 INFO Pod/odh-model-controller-6849fc49d8-nzgnp: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 816ms (816ms including waiting). Image size: 238959091 bytes. 2026/07/16 16:03:21 15:59:41 INFO Pod/odh-model-controller-6849fc49d8-nzgnp: Created - Created container: manager 2026/07/16 16:03:21 15:59:41 INFO Pod/odh-model-controller-6849fc49d8-nzgnp: Started - Started container manager 2026/07/16 16:03:21 15:59:41 INFO Pod/model-serving-api-6688cd56bd-jr8hz: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/16 16:03:21 15:59:41 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-jr8hz 2026/07/16 16:03:21 15:59:41 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/16 16:03:21 15:59:41 INFO Pod/model-serving-api-6688cd56bd-jr8hz: Started - Started container server 2026/07/16 16:03:21 15:59:41 INFO Pod/model-serving-api-6688cd56bd-jr8hz: Created - Created container: server 2026/07/16 16:03:21 15:59:40 INFO Pod/odh-model-controller-6849fc49d8-nzgnp: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/16 16:03:21 15:59:40 INFO Pod/maas-controller-85bdcf9988-7p5hx: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/16 16:03:21 15:59:40 INFO Pod/maas-controller-85bdcf9988-7p5hx: AddedInterface - Add eth0 [10.130.2.40/23] from ovn-kubernetes 2026/07/16 16:03:21 15:59:40 INFO Pod/odh-model-controller-6849fc49d8-nzgnp: AddedInterface - Add eth0 [10.130.2.39/23] from ovn-kubernetes 2026/07/16 16:03:21 15:59:39 INFO Pod/maas-controller-85bdcf9988-7p5hx: Scheduled - Successfully assigned redhat-ods-applications/maas-controller-85bdcf9988-7p5hx to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 16:03:21 15:59:39 INFO Pod/odh-model-controller-6849fc49d8-nzgnp: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-nzgnp to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 16:03:21 15:59:39 INFO ReplicaSet/maas-controller-85bdcf9988: SuccessfulCreate - Created pod: maas-controller-85bdcf9988-7p5hx 2026/07/16 16:03:21 15:59:39 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/16 16:03:21 15:59:39 INFO Deployment/maas-controller: ScalingReplicaSet - Scaled up replica set maas-controller-85bdcf9988 from 0 to 1 2026/07/16 16:03:21 15:59:39 WARN Pod/odh-model-controller-6849fc49d8-nzgnp: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/16 16:03:21 15:59:39 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-nzgnp 2026/07/16 16:03:21 15:59:28 INFO Pod/maas-postgres-d66f546cf-7n6m5: Pulled - Successfully pulled image "registry.redhat.io/rhel9/postgresql-15:latest" in 9.448s (9.448s including waiting). Image size: 400814167 bytes. 2026/07/16 16:03:21 15:59:28 INFO Pod/maas-postgres-d66f546cf-7n6m5: Created - Created container: postgres 2026/07/16 16:03:21 15:59:28 INFO Pod/maas-postgres-d66f546cf-7n6m5: Started - Started container postgres 2026/07/16 16:03:21 15:59:18 INFO Pod/maas-postgres-d66f546cf-7n6m5: Scheduled - Successfully assigned redhat-ods-applications/maas-postgres-d66f546cf-7n6m5 to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 16:03:21 15:59:18 INFO Pod/maas-postgres-d66f546cf-7n6m5: AddedInterface - Add eth0 [10.130.2.38/23] from ovn-kubernetes 2026/07/16 16:03:21 15:59:18 INFO ReplicaSet/maas-postgres-d66f546cf: SuccessfulCreate - Created pod: maas-postgres-d66f546cf-7n6m5 2026/07/16 16:03:21 15:59:18 INFO Pod/maas-postgres-d66f546cf-7n6m5: Pulling - Pulling image "registry.redhat.io/rhel9/postgresql-15:latest" 2026/07/16 16:03:21 15:59:18 INFO Deployment/maas-postgres: ScalingReplicaSet - Scaled up replica set maas-postgres-d66f546cf from 0 to 1 2026/07/16 16:03:21 15:59:08 INFO Pod/trustyai-service-operator-controller-manager-85df4c5659-vbdbr: Killing - Stopping container manager 2026/07/16 16:03:21 15:58:59 INFO Pod/maas-api-84449bdd5d-g7ch2: Killing - Stopping container maas-api 2026/07/16 16:03:21 15:58:59 INFO Lease/b7e9931f.trustyai.opendatahub.io: LeaderElection - trustyai-service-operator-controller-manager-85df4c5659-vbdbr_f91a104b-ce19-4618-9264-05c3c5450251 became leader 2026/07/16 16:03:21 15:58:58 INFO Pod/llmisvc-controller-manager-5d65df97bb-qkcf7: Killing - Stopping container manager 2026/07/16 16:03:21 15:58:58 INFO Pod/model-serving-api-6688cd56bd-55w6p: Killing - Stopping container server 2026/07/16 16:03:21 15:58:58 INFO Pod/odh-model-controller-6849fc49d8-47gsc: Killing - Stopping container manager 2026/07/16 16:03:21 15:58:58 INFO Pod/kserve-controller-manager-6c67dcb69c-ltjcl: Killing - Stopping container manager 2026/07/16 16:03:21 15:58:57 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 16:03:21 15:58:43 INFO Pod/trustyai-service-operator-controller-manager-85df4c5659-vbdbr: Started - Started container manager 2026/07/16 16:03:21 15:58:43 INFO Pod/trustyai-service-operator-controller-manager-85df4c5659-vbdbr: AddedInterface - Add eth0 [10.130.2.37/23] from ovn-kubernetes 2026/07/16 16:03:21 15:58:43 INFO Pod/trustyai-service-operator-controller-manager-85df4c5659-vbdbr: Pulled - Container image "registry.redhat.io/rhoai/odh-trustyai-service-operator-rhel9@sha256:17b36dfec55d1d9aeb3e778d1cf468b918e2b762efb53af1aa6af095387e4da2" already present on machine 2026/07/16 16:03:21 15:58:43 INFO Pod/trustyai-service-operator-controller-manager-85df4c5659-vbdbr: Created - Created container: manager 2026/07/16 16:03:21 15:58:42 INFO Pod/trustyai-service-operator-controller-manager-85df4c5659-vbdbr: Scheduled - Successfully assigned redhat-ods-applications/trustyai-service-operator-controller-manager-85df4c5659-vbdbr to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 16:03:21 15:58:42 INFO Pod/trustyai-service-operator-controller-manager-85df4c5659-tjgrd: Killing - Stopping container manager 2026/07/16 16:03:21 15:58:42 INFO Pod/trustyai-service-operator-controller-manager-85df4c5659-sbqjz: Killing - Stopping container manager 2026/07/16 16:03:21 15:58:42 INFO ReplicaSet/trustyai-service-operator-controller-manager-85df4c5659: SuccessfulCreate - Created pod: trustyai-service-operator-controller-manager-85df4c5659-vbdbr 2026/07/16 16:03:21 15:58:42 INFO Deployment/trustyai-service-operator-controller-manager: ScalingReplicaSet - Scaled up replica set trustyai-service-operator-controller-manager-85df4c5659 from 0 to 1 2026/07/16 16:03:21 === RESOURCE QUOTAS === 2026/07/16 16:03:21 No resource quotas found in monitored namespaces 2026/07/16 16:03:21 Classification for components: infrastructure/storage (code=1006, confidence=medium) 2026/07/16 16:03:21 Evidence: event Pod/odh-model-controller-6849fc49d8-wswvx: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/odh-model-controller-6849fc49d8-wswvx: FailedMount - MountVolume.SetUp failed for volume \"cert\" : secret \"odh-model-controller-webhook-cert\" not found"],"confidence":"medium"} 2026/07/16 16:03:21 Diagnostics complete for failed test: components controller_test.go:484: Stopping: components test failed. 2026/07/16 16:03:21 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/16 16:03:21 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/16 16:03:23 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (1045.79s)