=== RUN TestOdhOperator/components/group_2/kueue/Validate_component_removed_to_unmanaged_transition helper_test.go:150: Setting Kueue component (default-kueue) to Removed mode to start with a clean state. helper_test.go:150: Cleaning up any existing Kueue test resources. helper_test.go:150: Cleaning up Kueue cluster-scoped resources helper_test.go:150: Will attempt to delete 2 Kueue resources helper_test.go:150: Attempting to delete ClusterQueue /default helper_test.go:150: Successfully processed deletion of ClusterQueue /default helper_test.go:150: Attempting to delete Kueue /cluster helper_test.go:150: Successfully processed deletion of Kueue /cluster helper_test.go:150: Creating test namespace (test-kueue-managed-d9c4bevs84ic7b59vf90) with Kueue management annotation. helper_test.go:150: Creating Kueue ConfigMap (kueue-manager-config) to verify default Kueue resource is created correctly. helper_test.go:150: Creating Kueue ConfigMap (kueue-manager-config) in namespace (redhat-ods-applications). helper_test.go:150: Updating the management state of the component in the DataScienceCluster to Unmanaged (expecting NotReady initially due to missing operator). helper_test.go:150: Ensuring LWS Operator is installed (namespace=openshift-lws-operator, name=leader-worker-set). helper_test.go:150: Ensuring LeaderWorkerSetOperator CR exists (namespace=openshift-lws-operator, name=cluster). helper_test.go:150: Ensuring OCP Kueue Operator is installed (namespace=openshift-kueue-operator, name=kueue-operator). helper_test.go:150: Verifying the component in the DataScienceCluster transitions to Ready state. helper_test.go:150: Verifying Kueue configuration (cluster) is created with all expected frameworks. testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_2/kueue/Validate_component_removed_to_unmanaged_transition (30.70s) === RUN TestOdhOperator/components/group_2/kueue === PAUSE TestOdhOperator/components/group_2/kueue === CONT TestOdhOperator/components/group_2/kueue test_context_test.go:223: Timed out after 10.001s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:114 with: Expected resource 'cluster' of kind 'Kueue' to match condition {(["BatchJob", "Deployment", "JobSet", "LeaderWorkerSet", "MPIJob", "PaddleJob", "Pod", "PyTorchJob", "RayCluster", "RayJob", "StatefulSet", "TFJob", "TrainJob", "XGBoostJob"] - .spec.config.integrations.frameworks) | length == 0 []}. Expected <string>: map[apiVersion:kueue.openshift.io/v1 kind:Kueue metadata:map[annotations:map[platform.opendatahub.io/instance.generation:1 platform.opendatahub.io/instance.name:default-kueue platform.opendatahub.io/instance.uid:492b33b3-52b6-4947-851d-71fb2807ca19 platform.opendatahub.io/type:OpenShift AI Self-Managed platform.opendatahub.io/version:3.5.0-ea.2] creationTimestamp:2026-07-16T01:55:54Z deletionGracePeriodSeconds:0 deletionTimestamp:2026-07-16T02:45:25Z finalizers:[kueue.openshift.io/finalizer] generation:2 labels:map[platform.opendatahub.io/part-of:kueue] managedFields:[map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:status:map[f:conditions:map[k:{"type":"Available"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"CertManagerAvailable"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"Degraded"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"Progressing"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]]] f:readyReplicas:map[]]] manager:kueue-operator operation:Apply subresource:status time:2026-07-16T01:56:35Z] map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[.:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:labels:map[.:map[] f:platform.opendatahub.io/part-of:map[]]] f:spec:map[.:map[] f:config:map[.:map[] f:integrations:map[.:map[] f:frameworks:map[.:map[] v:"Deployment":map[] v:"Pod":map[] v:"PyTorchJob":map[] v:"RayCluster":map[] v:"RayJob":map[] v:"StatefulSet":map[] v:"TrainJob":map[]]]] f:logLevel:map[] f:managementState:map[] f:operatorLogLevel:map[]]] manager:manager operation:Update time:2026-07-16T01:55:54Z] map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:finalizers:map[.:map[] v:"kueue.openshift.io/finalizer":map[]]]] manager:kueue-operator operation:Update time:2026-07-16T02:45:25Z]] name:cluster resourceVersion:395553521 uid:1044231e-05a9-479e-8960-44e8963259bd] spec:map[config:map[integrations:map[frameworks:[Deployment Pod PyTorchJob RayCluster RayJob StatefulSet TrainJob]]] logLevel:Normal managementState:Managed operatorLogLevel:Normal] status:map[conditions:[map[lastTransitionTime:2026-07-16T01:56:35Z message:All 2 replicas are ready reason:AllReplicasReady status:True type:Available] map[lastTransitionTime:2026-07-16T01:56:35Z message:Deployment is up to date reason:AsExpected status:False type:Progressing] map[lastTransitionTime:2026-07-16T01:56:35Z message: reason:AsExpected status:False type:Degraded] map[lastTransitionTime:2026-07-16T01:56:20Z message:cert-manager is installed reason:CertManagerInstalled status:True type:CertManagerAvailable]] readyReplicas:2]] to match expression <string>: (["BatchJob", "Deployment", "JobSet", "LeaderWorkerSet", "MPIJob", "PaddleJob", "Pod", "PyTorchJob", "RayCluster", "RayJob", "StatefulSet", "TFJob", "TrainJob", "XGBoostJob"] - .spec.config.integrations.frameworks) | length == 0 test_context_test.go:1106: cleanup: best-effort delete failed for /, Kind= redhat-ods-applications/kueue-manager-config: Object 'Kind' is missing in 'unstructured object has no kind' --- FAIL: TestOdhOperator/components/group_2/kueue (30.95s) === RUN TestOdhOperator/components/group_2 --- FAIL: TestOdhOperator/components/group_2 (0.00s) === RUN TestOdhOperator/components 2026/07/16 02:48:57 TEST FAILURE DETECTED: group 2 2026/07/16 02:48:57 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 2) === 2026/07/16 02:48:57 === CLUSTER STATE === 2026/07/16 02:48:57 Node: ods-qe-psi-07-b4mqp-master-0 2026/07/16 02:48:57 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 02:48:57 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 02:48:57 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 02:48:57 Condition Ready: True - kubelet is posting ready status 2026/07/16 02:48:57 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/16 02:48:57 Node: ods-qe-psi-07-b4mqp-master-1 2026/07/16 02:48:57 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 02:48:57 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 02:48:57 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 02:48:57 Condition Ready: True - kubelet is posting ready status 2026/07/16 02:48:57 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/16 02:48:57 Node: ods-qe-psi-07-b4mqp-master-2 2026/07/16 02:48:57 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 02:48:57 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 02:48:57 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 02:48:57 Condition Ready: True - kubelet is posting ready status 2026/07/16 02:48:57 Resources: 11500m CPU, 23450732Ki memory allocatable (of 12000m CPU, 24601708Ki memory capacity) 2026/07/16 02:48:57 Node: ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/16 02:48:57 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 02:48:57 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 02:48:57 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 02:48:57 Condition Ready: True - kubelet is posting ready status 2026/07/16 02:48:57 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/16 02:48:57 Node: ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/16 02:48:57 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 02:48:57 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 02:48:57 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 02:48:57 Condition Ready: True - kubelet is posting ready status 2026/07/16 02:48:57 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/16 02:48:57 Node: ods-qe-psi-07-b4mqp-worker-0-jd4pm 2026/07/16 02:48:57 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 02:48:57 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 02:48:57 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 02:48:57 Condition Ready: True - kubelet is posting ready status 2026/07/16 02:48:57 Resources: 11500m CPU, 23452736Ki memory allocatable (of 12000m CPU, 24603712Ki memory capacity) 2026/07/16 02:48:57 Node: ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 02:48:57 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 02:48:57 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 02:48:57 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 02:48:57 Condition Ready: True - kubelet is posting ready status 2026/07/16 02:48:57 Resources: 11500m CPU, 23452744Ki memory allocatable (of 12000m CPU, 24603720Ki memory capacity) 2026/07/16 02:48:57 Node: ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/16 02:48:57 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 02:48:57 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 02:48:57 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 02:48:57 Condition Ready: True - kubelet is posting ready status 2026/07/16 02:48:57 Resources: 11500m CPU, 23450720Ki memory allocatable (of 12000m CPU, 24601696Ki memory capacity) 2026/07/16 02:48:57 === DEPLOYMENTS === 2026/07/16 02:48:57 Namespace: redhat-ods-applications 2026/07/16 02:48:57 Deployment maas-api: 1/1 (READY) 2026/07/16 02:48:57 Deployment maas-controller: 1/1 (READY) 2026/07/16 02:48:57 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/16 02:48:57 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/16 02:48:57 Namespace: kube-system 2026/07/16 02:48:57 === PODS === 2026/07/16 02:48:57 Namespace redhat-ods-applications: no problematic pods found 2026/07/16 02:48:57 Namespace kube-system: no problematic pods found 2026/07/16 02:48:57 === OPERATOR STATUS === 2026/07/16 02:48:57 Operator deployment: 3/3 ready 2026/07/16 02:48:57 === DSCI STATUS === 2026/07/16 02:48:57 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/16 02:48:57 === DSC STATUS === 2026/07/16 02:48:57 Failed to collect DSC data: default-dsc unhealthy conditions: ModelsAsServiceReady=False 2026/07/16 02:48:57 === RECENT EVENTS === 2026/07/16 02:48:57 02:48:49 INFO Pod/model-serving-api-6688cd56bd-zfg7k: Killing - Stopping container server 2026/07/16 02:48:57 02:48:49 INFO Pod/odh-model-controller-867855946f-2rm44: Killing - Stopping container manager 2026/07/16 02:48:57 02:48:49 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-dsphh: Killing - Stopping container manager 2026/07/16 02:48:57 02:48:49 INFO Pod/model-registry-operator-controller-manager-76b5b7f674-dj4bv: Killing - Stopping container manager 2026/07/16 02:48:57 02:48:48 INFO Pod/kserve-controller-manager-6c67dcb69c-66gh4: Killing - Stopping container manager 2026/07/16 02:48:57 02:48:48 INFO Pod/llmisvc-controller-manager-5d65df97bb-czktm: Killing - Stopping container manager 2026/07/16 02:48:57 02:48:48 WARN Endpoints/llmisvc-webhook-server-service: FailedToUpdateEndpoint - Failed to update endpoint redhat-ods-applications/llmisvc-webhook-server-service: Operation cannot be fulfilled on endpoints "llmisvc-webhook-server-service": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/llmisvc-webhook-server-service, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 3deed722-ff99-4b42-bde8-91283b92253e, UID in object meta: 2026/07/16 02:48:57 02:48:34 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-867855946f-2rm44_f2ffbe98-9649-44f3-a9e1-9b66309da258 became leader 2026/07/16 02:48:57 02:48:33 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-5bd688895b-dsphh_6730bf78-83c0-41b4-93e8-d77bffc1e9cf became leader 2026/07/16 02:48:57 02:48:32 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-dsphh: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-5bd688895b-dsphh to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/16 02:48:57 02:48:32 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-dsphh: Started - Started container manager 2026/07/16 02:48:57 02:48:32 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-5dtd8: Killing - Stopping container manager 2026/07/16 02:48:57 02:48:32 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-dsphh: AddedInterface - Add eth0 [10.131.1.207/23] from ovn-kubernetes 2026/07/16 02:48:57 02:48:32 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-dsphh: Pulled - Container image "registry.redhat.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" already present on machine 2026/07/16 02:48:57 02:48:32 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-dsphh: Created - Created container: manager 2026/07/16 02:48:57 02:48:32 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-5bd688895b-dsphh 2026/07/16 02:48:57 02:48:32 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-5bd688895b from 0 to 1 2026/07/16 02:48:57 02:48:32 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-2lxgh: Killing - Stopping container manager 2026/07/16 02:48:57 02:48:18 INFO Pod/odh-model-controller-867855946f-2rm44: Started - Started container manager 2026/07/16 02:48:57 02:48:18 INFO Pod/odh-model-controller-867855946f-2rm44: Created - Created container: manager 2026/07/16 02:48:57 02:48:18 INFO Pod/odh-model-controller-867855946f-2rm44: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 780ms (780ms including waiting). Image size: 238959091 bytes. 2026/07/16 02:48:57 02:48:17 INFO Pod/odh-model-controller-867855946f-2rm44: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-867855946f-2rm44 to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/16 02:48:57 02:48:17 INFO ReplicaSet/odh-model-controller-867855946f: SuccessfulCreate - Created pod: odh-model-controller-867855946f-2rm44 2026/07/16 02:48:57 02:48:17 INFO Pod/odh-model-controller-867855946f-spmn6: Killing - Stopping container manager 2026/07/16 02:48:57 02:48:17 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-867855946f from 0 to 1 2026/07/16 02:48:57 02:48:17 INFO Pod/odh-model-controller-867855946f-kngjc: Killing - Stopping container manager 2026/07/16 02:48:57 02:48:17 INFO Pod/odh-model-controller-867855946f-2rm44: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/16 02:48:57 02:48:17 INFO Pod/odh-model-controller-867855946f-2rm44: AddedInterface - Add eth0 [10.131.1.206/23] from ovn-kubernetes 2026/07/16 02:48:57 02:48:02 INFO Pod/model-serving-api-6688cd56bd-zfg7k: AddedInterface - Add eth0 [10.129.3.183/23] from ovn-kubernetes 2026/07/16 02:48:57 02:48:02 INFO Pod/model-serving-api-6688cd56bd-zfg7k: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/16 02:48:57 02:48:02 INFO Pod/model-serving-api-6688cd56bd-zfg7k: Created - Created container: server 2026/07/16 02:48:57 02:48:02 INFO Pod/model-serving-api-6688cd56bd-zfg7k: Started - Started container server 2026/07/16 02:48:57 02:48:01 INFO Pod/model-serving-api-6688cd56bd-zfg7k: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-zfg7k to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/16 02:48:57 02:48:01 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/16 02:48:57 02:48:01 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-zfg7k 2026/07/16 02:48:57 02:48:01 INFO Pod/model-serving-api-6688cd56bd-x8dqk: Killing - Stopping container server 2026/07/16 02:48:57 02:48:01 INFO Pod/model-serving-api-6688cd56bd-f7j9v: Killing - Stopping container server 2026/07/16 02:48:57 02:47:01 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-5dtd8: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-5bd688895b-5dtd8 to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/16 02:48:57 02:47:01 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-5dtd8: AddedInterface - Add eth0 [10.131.1.205/23] from ovn-kubernetes 2026/07/16 02:48:57 02:47:01 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-5dtd8: Pulled - Container image "registry.redhat.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" already present on machine 2026/07/16 02:48:57 02:47:01 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-5dtd8: Created - Created container: manager 2026/07/16 02:48:57 02:47:01 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-5dtd8: Started - Started container manager 2026/07/16 02:48:57 02:47:01 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-5bd688895b-5dtd8 2026/07/16 02:48:57 02:47:01 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-5bd688895b from 1 to 2 2026/07/16 02:48:57 02:46:42 INFO Pod/odh-model-controller-867855946f-spmn6: Started - Started container manager 2026/07/16 02:48:57 02:46:42 INFO Pod/odh-model-controller-867855946f-spmn6: Created - Created container: manager 2026/07/16 02:48:57 02:46:42 INFO Pod/odh-model-controller-867855946f-spmn6: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 809ms (809ms including waiting). Image size: 238959091 bytes. 2026/07/16 02:48:57 02:46:41 INFO Pod/odh-model-controller-867855946f-spmn6: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-867855946f-spmn6 to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/16 02:48:57 02:46:41 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-867855946f from 1 to 2 2026/07/16 02:48:57 02:46:41 INFO ReplicaSet/odh-model-controller-867855946f: SuccessfulCreate - Created pod: odh-model-controller-867855946f-spmn6 2026/07/16 02:48:57 02:46:41 INFO Pod/odh-model-controller-867855946f-spmn6: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/16 02:48:57 02:46:41 INFO Pod/odh-model-controller-867855946f-spmn6: AddedInterface - Add eth0 [10.129.3.182/23] from ovn-kubernetes 2026/07/16 02:48:57 02:46:21 INFO Pod/model-serving-api-6688cd56bd-x8dqk: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-x8dqk to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/16 02:48:57 02:46:21 INFO Pod/model-serving-api-6688cd56bd-x8dqk: Started - Started container server 2026/07/16 02:48:57 02:46:21 INFO Pod/model-serving-api-6688cd56bd-x8dqk: AddedInterface - Add eth0 [10.129.3.181/23] from ovn-kubernetes 2026/07/16 02:48:57 02:46:21 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 1 to 2 2026/07/16 02:48:57 02:46:21 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-x8dqk 2026/07/16 02:48:57 02:46:21 INFO Pod/model-serving-api-6688cd56bd-x8dqk: Created - Created container: server 2026/07/16 02:48:57 02:46:21 INFO Pod/model-serving-api-6688cd56bd-x8dqk: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/16 02:48:57 02:46:09 INFO Pod/maas-api-84449bdd5d-mfrt8: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" in 791ms (791ms including waiting). Image size: 175018396 bytes. 2026/07/16 02:48:57 02:46:09 INFO Pod/maas-api-84449bdd5d-mfrt8: Created - Created container: maas-api 2026/07/16 02:48:57 02:46:09 INFO Pod/maas-api-84449bdd5d-mfrt8: Started - Started container maas-api 2026/07/16 02:48:57 02:46:08 INFO Pod/maas-api-84449bdd5d-mfrt8: Scheduled - Successfully assigned redhat-ods-applications/maas-api-84449bdd5d-mfrt8 to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/16 02:48:57 02:46:08 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 02:48:57 02:46:08 INFO Pod/maas-api-84449bdd5d-mfrt8: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" 2026/07/16 02:48:57 02:46:08 INFO Pod/maas-api-84449bdd5d-mfrt8: AddedInterface - Add eth0 [10.129.3.178/23] from ovn-kubernetes 2026/07/16 02:48:57 02:46:08 WARN ReplicaSet/maas-api-84449bdd5d: FailedCreate - Error creating: pods "maas-api-84449bdd5d-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 2026/07/16 02:48:57 02:46:08 INFO ReplicaSet/maas-api-84449bdd5d: SuccessfulCreate - Created pod: maas-api-84449bdd5d-mfrt8 2026/07/16 02:48:57 02:46:07 INFO Deployment/maas-api: ScalingReplicaSet - Scaled up replica set maas-api-84449bdd5d from 0 to 1 2026/07/16 02:48:57 02:46:02 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-czktm_ff7d8ea1-1dc0-4ab7-82c7-8bec2d8097a0 became leader 2026/07/16 02:48:57 02:46:01 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-66gh4_df29f6a8-e427-4379-91ed-2aa22504205f became leader 2026/07/16 02:48:57 02:45:42 INFO Pod/llmisvc-controller-manager-5d65df97bb-czktm: Started - Started container manager 2026/07/16 02:48:57 02:45:42 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-2lxgh: Started - Started container manager 2026/07/16 02:48:57 02:45:42 INFO Pod/kserve-controller-manager-6c67dcb69c-66gh4: Created - Created container: manager 2026/07/16 02:48:57 02:45:42 INFO Pod/llmisvc-controller-manager-5d65df97bb-czktm: Created - Created container: manager 2026/07/16 02:48:57 02:45:42 INFO Pod/llmisvc-controller-manager-5d65df97bb-czktm: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 972ms (972ms including waiting). Image size: 236366599 bytes. 2026/07/16 02:48:57 02:45:42 INFO Pod/kserve-controller-manager-6c67dcb69c-66gh4: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 795ms (795ms including waiting). Image size: 238435613 bytes. 2026/07/16 02:48:57 02:45:42 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-2lxgh: Created - Created container: manager 2026/07/16 02:48:57 02:45:42 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-5bd688895b-2lxgh_66c0ad09-3387-4454-aad8-21f757cecd40 became leader 2026/07/16 02:48:57 02:45:42 INFO Pod/kserve-controller-manager-6c67dcb69c-66gh4: Started - Started container manager 2026/07/16 02:48:57 02:45:41 INFO Pod/llmisvc-controller-manager-5d65df97bb-czktm: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-czktm to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/16 02:48:57 02:45:41 INFO Pod/kserve-controller-manager-6c67dcb69c-66gh4: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-66gh4 to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/16 02:48:57 02:45:41 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-czktm 2026/07/16 02:48:57 02:45:41 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/16 02:48:57 02:45:41 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-66gh4 2026/07/16 02:48:57 02:45:41 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-2lxgh: AddedInterface - Add eth0 [10.129.3.175/23] from ovn-kubernetes 2026/07/16 02:48:57 02:45:41 INFO Pod/kserve-controller-manager-6c67dcb69c-66gh4: AddedInterface - Add eth0 [10.129.3.176/23] from ovn-kubernetes 2026/07/16 02:48:57 02:45:41 INFO Pod/llmisvc-controller-manager-5d65df97bb-czktm: AddedInterface - Add eth0 [10.131.1.203/23] from ovn-kubernetes 2026/07/16 02:48:57 02:45:41 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-2lxgh: Pulled - Container image "registry.redhat.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" already present on machine 2026/07/16 02:48:57 02:45:41 INFO Pod/kserve-controller-manager-6c67dcb69c-66gh4: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/16 02:48:57 02:45:41 INFO Pod/llmisvc-controller-manager-5d65df97bb-czktm: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/16 02:48:57 02:45:41 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/16 02:48:57 02:45:40 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-2lxgh: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-5bd688895b-2lxgh to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/16 02:48:57 02:45:40 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-5bd688895b from 0 to 1 2026/07/16 02:48:57 02:45:40 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-5bd688895b-2lxgh 2026/07/16 02:48:57 02:45:40 WARN ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: FailedCreate - Error creating: pods "workload-variant-autoscaler-controller-manager-5bd688895b-" is forbidden: error looking up service account redhat-ods-applications/workload-variant-autoscaler-controller-manager: serviceaccount "workload-variant-autoscaler-controller-manager" not found 2026/07/16 02:48:57 02:45:40 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-867855946f-kngjc_7db540f3-f13b-40f2-b315-9a1bc4ea1bde became leader 2026/07/16 02:48:57 02:45:40 WARN Pod/workload-variant-autoscaler-controller-manager-5bd688895b-2lxgh: FailedMount - MountVolume.SetUp failed for volume "epp-metrics-token" : secret "workload-variant-autoscaler-epp-metrics-token" not found 2026/07/16 02:48:57 02:45:40 WARN ServiceMonitor/workload-variant-autoscaler-controller-manager-metrics-monitor: InvalidConfiguration - ServiceMonitor workload-variant-autoscaler-controller-manager-metrics-monitor was rejected due to invalid configuration: unable to get secret "workload-variant-autoscaler-controller-manager-token": secrets "workload-variant-autoscaler-controller-manager-token" not found 2026/07/16 02:48:57 02:45:38 INFO Lease/85f368d1.opendatahub.io: LeaderElection - model-registry-operator-controller-manager-76b5b7f674-dj4bv_52a9b8b4-d8a8-40c9-b2df-fb6027d14773 became leader 2026/07/16 02:48:57 02:45:36 INFO Lease/maas-controller.models-as-a-service.opendatahub.io: LeaderElection - maas-controller-85bdcf9988-nf8dd_9b17ee9e-5565-440e-8a70-ee600ca43af6 became leader 2026/07/16 02:48:57 02:45:23 INFO Pod/model-registry-operator-controller-manager-76b5b7f674-dj4bv: Started - Started container manager 2026/07/16 02:48:57 02:45:23 INFO Pod/model-registry-operator-controller-manager-76b5b7f674-dj4bv: Created - Created container: manager 2026/07/16 02:48:57 02:45:23 INFO Pod/model-registry-operator-controller-manager-76b5b7f674-dj4bv: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-registry-operator-rhel9@sha256:ebcec5c43468283ed3cbab50530f939166f8f3e8c9d306e91a8107d05d7fa9d4" in 3.567s (3.567s including waiting). Image size: 211356146 bytes. 2026/07/16 02:48:57 02:45:22 INFO Pod/model-serving-api-6688cd56bd-f7j9v: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/16 02:48:57 02:45:22 INFO Pod/model-serving-api-6688cd56bd-f7j9v: Started - Started container server 2026/07/16 02:48:57 02:45:22 INFO Pod/model-serving-api-6688cd56bd-f7j9v: Created - Created container: server 2026/07/16 02:48:57 02:45:21 INFO Pod/model-serving-api-6688cd56bd-f7j9v: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-f7j9v to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/16 02:48:57 02:45:21 INFO Pod/odh-model-controller-867855946f-kngjc: Created - Created container: manager 2026/07/16 02:48:57 02:45:21 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/16 02:48:57 02:45:21 INFO Pod/model-serving-api-6688cd56bd-f7j9v: AddedInterface - Add eth0 [10.131.1.202/23] from ovn-kubernetes 2026/07/16 02:48:57 02:45:21 INFO Pod/odh-model-controller-867855946f-kngjc: Started - Started container manager 2026/07/16 02:48:57 02:45:21 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/16 02:48:57 02:45:21 INFO Pod/odh-model-controller-867855946f-kngjc: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 717ms (717ms including waiting). Image size: 238959091 bytes. 2026/07/16 02:48:57 02:45:21 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-f7j9v 2026/07/16 02:48:57 02:45:20 INFO Pod/odh-model-controller-867855946f-kngjc: AddedInterface - Add eth0 [10.131.1.201/23] from ovn-kubernetes 2026/07/16 02:48:57 02:45:20 INFO Pod/odh-model-controller-867855946f-kngjc: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/16 02:48:57 02:45:19 INFO Pod/odh-model-controller-867855946f-kngjc: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-867855946f-kngjc to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/16 02:48:57 02:45:19 INFO Pod/maas-controller-85bdcf9988-nf8dd: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 803ms (803ms including waiting). Image size: 212866077 bytes. 2026/07/16 02:48:57 02:45:19 INFO ReplicaSet/odh-model-controller-867855946f: SuccessfulCreate - Created pod: odh-model-controller-867855946f-kngjc 2026/07/16 02:48:57 02:45:19 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-867855946f from 0 to 1 2026/07/16 02:48:57 02:45:19 INFO Pod/model-registry-operator-controller-manager-76b5b7f674-dj4bv: AddedInterface - Add eth0 [10.131.1.200/23] from ovn-kubernetes 2026/07/16 02:48:57 02:45:19 INFO Pod/model-registry-operator-controller-manager-76b5b7f674-dj4bv: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-registry-operator-rhel9@sha256:ebcec5c43468283ed3cbab50530f939166f8f3e8c9d306e91a8107d05d7fa9d4" 2026/07/16 02:48:57 02:45:19 INFO Pod/maas-controller-85bdcf9988-nf8dd: Started - Started container manager 2026/07/16 02:48:57 02:45:19 WARN Pod/odh-model-controller-867855946f-kngjc: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/16 02:48:57 02:45:19 INFO Pod/maas-controller-85bdcf9988-nf8dd: Created - Created container: manager 2026/07/16 02:48:57 02:45:18 INFO Pod/model-registry-operator-controller-manager-76b5b7f674-dj4bv: Scheduled - Successfully assigned redhat-ods-applications/model-registry-operator-controller-manager-76b5b7f674-dj4bv to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/16 02:48:57 02:45:18 INFO Pod/maas-controller-85bdcf9988-nf8dd: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/16 02:48:57 02:45:18 INFO Deployment/model-registry-operator-controller-manager: ScalingReplicaSet - Scaled up replica set model-registry-operator-controller-manager-76b5b7f674 from 0 to 1 2026/07/16 02:48:57 02:45:18 INFO ReplicaSet/model-registry-operator-controller-manager-76b5b7f674: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-76b5b7f674-dj4bv 2026/07/16 02:48:57 02:45:18 WARN Pod/model-registry-operator-controller-manager-76b5b7f674-dj4bv: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 2026/07/16 02:48:57 02:45:18 INFO Pod/maas-controller-85bdcf9988-nf8dd: AddedInterface - Add eth0 [10.131.1.199/23] from ovn-kubernetes 2026/07/16 02:48:57 02:45:17 INFO Pod/maas-controller-85bdcf9988-nf8dd: Scheduled - Successfully assigned redhat-ods-applications/maas-controller-85bdcf9988-nf8dd to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/16 02:48:57 02:45:17 INFO ReplicaSet/maas-controller-85bdcf9988: SuccessfulCreate - Created pod: maas-controller-85bdcf9988-nf8dd 2026/07/16 02:48:57 02:45:17 INFO Deployment/maas-controller: ScalingReplicaSet - Scaled up replica set maas-controller-85bdcf9988 from 0 to 1 2026/07/16 02:48:57 02:45:06 INFO Pod/spark-operator-controller-767899d44f-d6srz: Killing - Stopping container controller 2026/07/16 02:48:57 02:45:06 INFO Pod/spark-operator-webhook-9cb9bdf44-4bxzl: Killing - Stopping container webhook 2026/07/16 02:48:57 02:45:05 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - ServiceMonitor controller-manager-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 02:48:57 02:44:51 INFO Lease/spark-operator-controller-lock: LeaderElection - spark-operator-controller-767899d44f-d6srz_2a4cfbb7-fe29-407d-a020-78b39b73587b became leader 2026/07/16 02:48:57 02:44:50 INFO Pod/spark-operator-webhook-9cb9bdf44-4bxzl: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-webhook-9cb9bdf44-4bxzl to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/16 02:48:57 02:44:50 INFO Pod/spark-operator-webhook-9cb9bdf44-4bxzl: Started - Started container webhook 2026/07/16 02:48:57 02:44:50 INFO Pod/spark-operator-webhook-9cb9bdf44-4bxzl: AddedInterface - Add eth0 [10.131.1.198/23] from ovn-kubernetes 2026/07/16 02:48:57 02:44:50 INFO ReplicaSet/spark-operator-webhook-9cb9bdf44: SuccessfulCreate - Created pod: spark-operator-webhook-9cb9bdf44-4bxzl 2026/07/16 02:48:57 02:44:50 INFO Pod/spark-operator-webhook-9cb9bdf44-4bxzl: Pulled - Container image "registry.redhat.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/16 02:48:57 02:44:50 INFO Pod/spark-operator-webhook-9cb9bdf44-4bxzl: Created - Created container: webhook 2026/07/16 02:48:57 02:44:49 INFO Deployment/spark-operator-webhook: ScalingReplicaSet - Scaled up replica set spark-operator-webhook-9cb9bdf44 from 0 to 1 2026/07/16 02:48:57 02:44:49 INFO Pod/spark-operator-webhook-9cb9bdf44-52wqf: Killing - Stopping container webhook 2026/07/16 02:48:57 02:44:49 INFO Pod/spark-operator-webhook-9cb9bdf44-6htzq: Killing - Stopping container webhook 2026/07/16 02:48:57 02:44:35 INFO Pod/spark-operator-controller-767899d44f-d6srz: AddedInterface - Add eth0 [10.131.1.197/23] from ovn-kubernetes 2026/07/16 02:48:57 02:44:35 INFO Pod/spark-operator-controller-767899d44f-d6srz: Started - Started container controller 2026/07/16 02:48:57 02:44:35 INFO Pod/spark-operator-controller-767899d44f-d6srz: Pulled - Container image "registry.redhat.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/16 02:48:57 02:44:35 INFO Pod/spark-operator-controller-767899d44f-d6srz: Created - Created container: controller 2026/07/16 02:48:57 02:44:34 INFO Pod/spark-operator-controller-767899d44f-d6srz: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-controller-767899d44f-d6srz to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/16 02:48:57 02:44:34 INFO Pod/spark-operator-controller-767899d44f-pqnwj: Killing - Stopping container controller 2026/07/16 02:48:57 02:44:34 INFO ReplicaSet/spark-operator-controller-767899d44f: SuccessfulCreate - Created pod: spark-operator-controller-767899d44f-d6srz 2026/07/16 02:48:57 02:44:34 INFO Deployment/spark-operator-controller: ScalingReplicaSet - Scaled up replica set spark-operator-controller-767899d44f from 0 to 1 2026/07/16 02:48:57 02:44:34 INFO Pod/spark-operator-controller-767899d44f-4nmhl: Killing - Stopping container controller 2026/07/16 02:48:57 02:44:15 INFO SparkApplication/scheduled-spark-pi-d9c4advs84ic7b59vf8g-1784169813000833080: SparkDriverCompleted - Driver scheduled-spark-pi-d9c4advs84ic7b59vf8g-1784169813000833080-driver completed 2026/07/16 02:48:57 02:44:12 INFO Pod/spark-pi-3441149f68cf25e7-exec-1: Killing - Stopping container spark-kubernetes-executor 2026/07/16 02:48:57 02:44:12 INFO SparkApplication/scheduled-spark-pi-d9c4advs84ic7b59vf8g-1784169813000833080: SparkExecutorCompleted - Executor spark-pi-3441149f68cf25e7-exec-1 completed 2026/07/16 02:48:57 02:43:58 INFO Pod/spark-pi-3441149f68cf25e7-exec-1: Started - Started container spark-kubernetes-executor 2026/07/16 02:48:57 02:43:58 INFO Pod/spark-pi-3441149f68cf25e7-exec-1: AddedInterface - Add eth0 [10.130.3.161/23] from ovn-kubernetes 2026/07/16 02:48:57 02:43:58 INFO SparkApplication/scheduled-spark-pi-d9c4advs84ic7b59vf8g-1784169813000833080: SparkExecutorRunning - Executor spark-pi-3441149f68cf25e7-exec-1 is running 2026/07/16 02:48:57 02:43:58 INFO Pod/spark-pi-3441149f68cf25e7-exec-1: Created - Created container: spark-kubernetes-executor 2026/07/16 02:48:57 02:43:58 INFO Pod/spark-pi-3441149f68cf25e7-exec-1: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/16 02:48:57 02:43:57 INFO Pod/spark-pi-3441149f68cf25e7-exec-1: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-3441149f68cf25e7-exec-1 to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 02:48:57 === RESOURCE QUOTAS === 2026/07/16 02:48:57 No resource quotas found in monitored namespaces 2026/07/16 02:48:57 Classification for group 2: infrastructure/storage (code=1006, confidence=medium) 2026/07/16 02:48:57 Evidence: event Pod/workload-variant-autoscaler-controller-manager-5bd688895b-2lxgh: FailedMount - MountVolume.SetUp failed for volume "epp-metrics-token" : secret "workload-variant-autoscaler-epp-metrics-token" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/workload-variant-autoscaler-controller-manager-5bd688895b-2lxgh: FailedMount - MountVolume.SetUp failed for volume \"epp-metrics-token\" : secret \"workload-variant-autoscaler-epp-metrics-token\" not found"],"confidence":"medium"} 2026/07/16 02:48:57 Diagnostics complete for failed test: group 2 controller_test.go:753: Stopping: group 2 test failed. 2026/07/16 02:48:57 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/16 02:48:57 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/16 02:53:16 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (1101.18s) === RUN TestOdhOperator 2026/07/16 02:38:58 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/16 02:57:30 TEST FAILURE DETECTED: components 2026/07/16 02:57:30 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/16 02:57:31 === CLUSTER STATE === 2026/07/16 02:57:31 Node: ods-qe-psi-07-b4mqp-master-0 2026/07/16 02:57:31 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 02:57:31 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 02:57:31 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 02:57:31 Condition Ready: True - kubelet is posting ready status 2026/07/16 02:57:31 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/16 02:57:31 Node: ods-qe-psi-07-b4mqp-master-1 2026/07/16 02:57:31 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 02:57:31 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 02:57:31 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 02:57:31 Condition Ready: True - kubelet is posting ready status 2026/07/16 02:57:31 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/16 02:57:31 Node: ods-qe-psi-07-b4mqp-master-2 2026/07/16 02:57:31 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 02:57:31 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 02:57:31 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 02:57:31 Condition Ready: True - kubelet is posting ready status 2026/07/16 02:57:31 Resources: 11500m CPU, 23450732Ki memory allocatable (of 12000m CPU, 24601708Ki memory capacity) 2026/07/16 02:57:31 Node: ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/16 02:57:31 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 02:57:31 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 02:57:31 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 02:57:31 Condition Ready: True - kubelet is posting ready status 2026/07/16 02:57:31 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/16 02:57:31 Node: ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/16 02:57:31 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 02:57:31 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 02:57:31 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 02:57:31 Condition Ready: True - kubelet is posting ready status 2026/07/16 02:57:31 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/16 02:57:31 Node: ods-qe-psi-07-b4mqp-worker-0-jd4pm 2026/07/16 02:57:31 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 02:57:31 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 02:57:31 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 02:57:31 Condition Ready: True - kubelet is posting ready status 2026/07/16 02:57:31 Resources: 11500m CPU, 23452736Ki memory allocatable (of 12000m CPU, 24603712Ki memory capacity) 2026/07/16 02:57:31 Node: ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/16 02:57:31 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 02:57:31 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 02:57:31 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 02:57:31 Condition Ready: True - kubelet is posting ready status 2026/07/16 02:57:31 Resources: 11500m CPU, 23452744Ki memory allocatable (of 12000m CPU, 24603720Ki memory capacity) 2026/07/16 02:57:31 Node: ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/16 02:57:31 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 02:57:31 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 02:57:31 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 02:57:31 Condition Ready: True - kubelet is posting ready status 2026/07/16 02:57:31 Resources: 11500m CPU, 23450720Ki memory allocatable (of 12000m CPU, 24601696Ki memory capacity) 2026/07/16 02:57:31 === DEPLOYMENTS === 2026/07/16 02:57:31 Namespace: kube-system 2026/07/16 02:57:31 Namespace: redhat-ods-applications 2026/07/16 02:57:31 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/16 02:57:31 Deployment kubeflow-trainer-controller-manager: 1/1 (READY) 2026/07/16 02:57:31 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/16 02:57:31 Deployment maas-postgres: 1/1 (READY) 2026/07/16 02:57:31 Deployment model-serving-api: 1/1 (READY) 2026/07/16 02:57:31 Deployment odh-model-controller: 1/1 (READY) 2026/07/16 02:57:31 === PODS === 2026/07/16 02:57:31 Namespace redhat-ods-applications: no problematic pods found 2026/07/16 02:57:31 Namespace kube-system: no problematic pods found 2026/07/16 02:57:31 === OPERATOR STATUS === 2026/07/16 02:57:31 Operator deployment: 3/3 ready 2026/07/16 02:57:31 === DSCI STATUS === 2026/07/16 02:57:31 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/16 02:57:31 === DSC STATUS === 2026/07/16 02:57:31 DSC default-dsc: 2026/07/16 02:57:31 Ready: True - 2026/07/16 02:57:31 ProvisioningSucceeded: True - 2026/07/16 02:57:31 ComponentsReady: True - 2026/07/16 02:57:31 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/16 02:57:31 DashboardReady: False - Component ManagementState is set to Removed 2026/07/16 02:57:31 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/16 02:57:31 KserveLLMInferenceServiceDependencies: True - 2026/07/16 02:57:31 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/16 02:57:31 KserveReady: True - 2026/07/16 02:57:31 KueueReady: True - 2026/07/16 02:57:31 LLM-D-WVADependencies: True - 2026/07/16 02:57:31 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/16 02:57:31 ModelControllerReady: True - 2026/07/16 02:57:31 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/16 02:57:31 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/16 02:57:31 OGXReady: False - Component ManagementState is set to Removed 2026/07/16 02:57:31 RayReady: False - Component ManagementState is set to Removed 2026/07/16 02:57:31 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/16 02:57:31 TrainerReady: True - 2026/07/16 02:57:31 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/16 02:57:31 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/16 02:57:31 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/16 02:57:31 === RECENT EVENTS === 2026/07/16 02:57:31 02:57:06 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-bd688_386e0eb3-16f3-4c33-86d1-f9d38e6da088 became leader 2026/07/16 02:57:31 02:57:05 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-mzj8k_0eed587d-8d0c-4710-9c3d-14cfbd7c4ff6 became leader 2026/07/16 02:57:31 02:56:50 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-dtqt4_1cc5cb13-8e19-4cf1-bae0-4fd16b148631 became leader 2026/07/16 02:57:31 02:56:48 INFO Pod/llmisvc-controller-manager-5d65df97bb-bd688: Started - Started container manager 2026/07/16 02:57:31 02:56:48 INFO Pod/llmisvc-controller-manager-5d65df97bb-bd688: Created - Created container: manager 2026/07/16 02:57:31 02:56:48 INFO Pod/llmisvc-controller-manager-5d65df97bb-bd688: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 780ms (780ms including waiting). Image size: 236366599 bytes. 2026/07/16 02:57:31 02:56:48 INFO Pod/kserve-controller-manager-6c67dcb69c-mzj8k: Started - Started container manager 2026/07/16 02:57:31 02:56:48 INFO Pod/kserve-controller-manager-6c67dcb69c-mzj8k: Created - Created container: manager 2026/07/16 02:57:31 02:56:48 INFO Pod/kserve-controller-manager-6c67dcb69c-mzj8k: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 732ms (732ms including waiting). Image size: 238435613 bytes. 2026/07/16 02:57:31 02:56:47 INFO Pod/llmisvc-controller-manager-5d65df97bb-bd688: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-bd688 to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/16 02:57:31 02:56:47 INFO Pod/kserve-controller-manager-6c67dcb69c-mzj8k: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-mzj8k to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/16 02:57:31 02:56:47 INFO Pod/kserve-controller-manager-6c67dcb69c-mzj8k: AddedInterface - Add eth0 [10.129.3.198/23] from ovn-kubernetes 2026/07/16 02:57:31 02:56:47 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/16 02:57:31 02:56:47 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/16 02:57:31 02:56:47 INFO Pod/llmisvc-controller-manager-5d65df97bb-bd688: AddedInterface - Add eth0 [10.131.1.222/23] from ovn-kubernetes 2026/07/16 02:57:31 02:56:47 INFO Pod/llmisvc-controller-manager-5d65df97bb-bd688: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/16 02:57:31 02:56:47 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-mzj8k 2026/07/16 02:57:31 02:56:47 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-bd688 2026/07/16 02:57:31 02:56:47 INFO Pod/kserve-controller-manager-6c67dcb69c-mzj8k: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/16 02:57:31 02:56:33 INFO Pod/model-serving-api-6688cd56bd-vlzxh: Created - Created container: server 2026/07/16 02:57:31 02:56:33 INFO Pod/model-serving-api-6688cd56bd-vlzxh: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/16 02:57:31 02:56:33 INFO Pod/model-serving-api-6688cd56bd-vlzxh: AddedInterface - Add eth0 [10.131.1.221/23] from ovn-kubernetes 2026/07/16 02:57:31 02:56:33 INFO Pod/model-serving-api-6688cd56bd-vlzxh: Started - Started container server 2026/07/16 02:57:31 02:56:32 INFO Pod/model-serving-api-6688cd56bd-vlzxh: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-vlzxh to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/16 02:57:31 02:56:32 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 02:57:31 02:56:32 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/16 02:57:31 02:56:32 WARN Pod/model-serving-api-6688cd56bd-vlzxh: FailedMount - MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 2026/07/16 02:57:31 02:56:32 INFO Pod/odh-model-controller-6849fc49d8-dtqt4: Started - Started container manager 2026/07/16 02:57:31 02:56:32 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-vlzxh 2026/07/16 02:57:31 02:56:32 INFO Pod/odh-model-controller-6849fc49d8-dtqt4: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 760ms (760ms including waiting). Image size: 238959091 bytes. 2026/07/16 02:57:31 02:56:32 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/16 02:57:31 02:56:32 INFO Pod/odh-model-controller-6849fc49d8-dtqt4: Created - Created container: manager 2026/07/16 02:57:31 02:56:31 INFO Pod/odh-model-controller-6849fc49d8-dtqt4: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/16 02:57:31 02:56:31 INFO Pod/odh-model-controller-6849fc49d8-dtqt4: AddedInterface - Add eth0 [10.129.3.197/23] from ovn-kubernetes 2026/07/16 02:57:31 02:56:30 INFO Pod/odh-model-controller-6849fc49d8-dtqt4: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-dtqt4 to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/16 02:57:31 02:56:30 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-dtqt4 2026/07/16 02:57:31 02:56:30 WARN Pod/odh-model-controller-6849fc49d8-dtqt4: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/16 02:57:31 02:56:30 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/16 02:57:31 02:56:20 INFO Pod/kserve-controller-manager-6c67dcb69c-7g86h: Killing - Stopping container manager 2026/07/16 02:57:31 02:56:20 INFO Pod/llmisvc-controller-manager-5d65df97bb-8smf9: Killing - Stopping container manager 2026/07/16 02:57:31 02:56:20 INFO Pod/odh-model-controller-6849fc49d8-b7sjk: Killing - Stopping container manager 2026/07/16 02:57:31 02:56:20 INFO Pod/model-serving-api-6688cd56bd-f8dw9: Killing - Stopping container server 2026/07/16 02:57:31 02:55:57 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-8smf9_bf6d7226-a6e7-48ab-ab1b-d9dccd9053d9 became leader 2026/07/16 02:57:31 02:55:55 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-7g86h_a9d4f995-761f-437a-8d90-8a7da51a2d43 became leader 2026/07/16 02:57:31 02:55:37 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-b7sjk_8c410a72-51b7-4dae-8b78-f033568335a4 became leader 2026/07/16 02:57:31 02:55:37 INFO Pod/llmisvc-controller-manager-5d65df97bb-8smf9: Started - Started container manager 2026/07/16 02:57:31 02:55:37 INFO Pod/llmisvc-controller-manager-5d65df97bb-8smf9: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 767ms (767ms including waiting). Image size: 236366599 bytes. 2026/07/16 02:57:31 02:55:37 INFO Pod/kserve-controller-manager-6c67dcb69c-7g86h: Started - Started container manager 2026/07/16 02:57:31 02:55:37 INFO Pod/kserve-controller-manager-6c67dcb69c-7g86h: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 837ms (837ms including waiting). Image size: 238435613 bytes. 2026/07/16 02:57:31 02:55:37 INFO Pod/kserve-controller-manager-6c67dcb69c-7g86h: Created - Created container: manager 2026/07/16 02:57:31 02:55:37 INFO Pod/llmisvc-controller-manager-5d65df97bb-8smf9: Created - Created container: manager 2026/07/16 02:57:31 02:55:36 INFO Pod/llmisvc-controller-manager-5d65df97bb-8smf9: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-8smf9 to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/16 02:57:31 02:55:36 INFO Pod/kserve-controller-manager-6c67dcb69c-7g86h: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-7g86h to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/16 02:57:31 02:55:36 INFO Pod/llmisvc-controller-manager-5d65df97bb-8smf9: AddedInterface - Add eth0 [10.131.1.220/23] from ovn-kubernetes 2026/07/16 02:57:31 02:55:36 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-8smf9 2026/07/16 02:57:31 02:55:36 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-7g86h 2026/07/16 02:57:31 02:55:36 INFO Pod/kserve-controller-manager-6c67dcb69c-7g86h: AddedInterface - Add eth0 [10.129.3.196/23] from ovn-kubernetes 2026/07/16 02:57:31 02:55:36 INFO Pod/kserve-controller-manager-6c67dcb69c-7g86h: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/16 02:57:31 02:55:36 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/16 02:57:31 02:55:36 INFO Pod/llmisvc-controller-manager-5d65df97bb-8smf9: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/16 02:57:31 02:55:36 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/16 02:57:31 02:55:32 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-7ccfdd66-v8557_154804e7-fc42-4b24-8edd-9cc8ddc3de61 became leader 2026/07/16 02:57:31 02:55:21 INFO Pod/model-serving-api-6688cd56bd-f8dw9: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-f8dw9 to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/16 02:57:31 02:55:21 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-f8dw9 2026/07/16 02:57:31 02:55:21 INFO Pod/odh-model-controller-6849fc49d8-b7sjk: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 639ms (639ms including waiting). Image size: 238959091 bytes. 2026/07/16 02:57:31 02:55:21 INFO Pod/model-serving-api-6688cd56bd-f8dw9: Created - Created container: server 2026/07/16 02:57:31 02:55:21 INFO Pod/odh-model-controller-6849fc49d8-b7sjk: Created - Created container: manager 2026/07/16 02:57:31 02:55:21 INFO Pod/model-serving-api-6688cd56bd-f8dw9: Started - Started container server 2026/07/16 02:57:31 02:55:21 INFO Pod/odh-model-controller-6849fc49d8-b7sjk: Started - Started container manager 2026/07/16 02:57:31 02:55:21 INFO Pod/model-serving-api-6688cd56bd-f8dw9: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/16 02:57:31 02:55:21 INFO Pod/model-serving-api-6688cd56bd-f8dw9: AddedInterface - Add eth0 [10.131.1.219/23] from ovn-kubernetes 2026/07/16 02:57:31 02:55:21 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 02:57:31 02:55:20 INFO Pod/odh-model-controller-6849fc49d8-b7sjk: AddedInterface - Add eth0 [10.129.3.195/23] from ovn-kubernetes 2026/07/16 02:57:31 02:55:20 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/16 02:57:31 02:55:20 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/16 02:57:31 02:55:20 INFO Pod/odh-model-controller-6849fc49d8-b7sjk: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/16 02:57:31 02:55:19 INFO Pod/odh-model-controller-6849fc49d8-b7sjk: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-b7sjk to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/16 02:57:31 02:55:19 INFO Pod/kserve-controller-manager-6c67dcb69c-fdkxf: Created - Created container: manager 2026/07/16 02:57:31 02:55:19 INFO Pod/kserve-controller-manager-6c67dcb69c-fdkxf: Started - Started container manager 2026/07/16 02:57:31 02:55:19 INFO Pod/kserve-controller-manager-6c67dcb69c-fdkxf: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 667ms (667ms including waiting). Image size: 238435613 bytes. 2026/07/16 02:57:31 02:55:19 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-b7sjk 2026/07/16 02:57:31 02:55:19 WARN Pod/llmisvc-controller-manager-5d65df97bb-mrtbx: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 2026/07/16 02:57:31 02:55:19 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/16 02:57:31 02:55:18 INFO Pod/kserve-controller-manager-6c67dcb69c-fdkxf: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/16 02:57:31 02:55:18 INFO Pod/kserve-controller-manager-6c67dcb69c-fdkxf: AddedInterface - Add eth0 [10.129.3.194/23] from ovn-kubernetes 2026/07/16 02:57:31 02:55:17 INFO Pod/llmisvc-controller-manager-5d65df97bb-mrtbx: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-mrtbx to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/16 02:57:31 02:55:17 INFO Pod/kserve-controller-manager-6c67dcb69c-fdkxf: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-fdkxf to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/16 02:57:31 02:55:17 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-v8557: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-trainer-rhel9@sha256:59ea523e2df9c1e705c755308ac11420682081cc63d54890e2b15fbd7d8ba570" in 1.81s (1.81s including waiting). Image size: 157432248 bytes. 2026/07/16 02:57:31 02:55:17 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/16 02:57:31 02:55:17 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-mrtbx 2026/07/16 02:57:31 02:55:17 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-fdkxf 2026/07/16 02:57:31 02:55:17 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/16 02:57:31 02:55:17 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-v8557: Created - Created container: manager 2026/07/16 02:57:31 02:55:17 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-v8557: Started - Started container manager 2026/07/16 02:57:31 02:55:15 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-v8557: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-7ccfdd66-v8557 to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/16 02:57:31 02:55:15 INFO ReplicaSet/kubeflow-trainer-controller-manager-7ccfdd66: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-7ccfdd66-v8557 2026/07/16 02:57:31 02:55:15 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-v8557: AddedInterface - Add eth0 [10.131.1.217/23] from ovn-kubernetes 2026/07/16 02:57:31 02:55:15 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-v8557: Pulling - Pulling image "registry.redhat.io/rhoai/odh-trainer-rhel9@sha256:59ea523e2df9c1e705c755308ac11420682081cc63d54890e2b15fbd7d8ba570" 2026/07/16 02:57:31 02:55:14 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-7ccfdd66 from 0 to 1 2026/07/16 02:57:31 02:55:14 WARN Pod/kserve-controller-manager-6c67dcb69c-l8cmz: Unhealthy - Readiness probe failed: Get "http://10.131.1.215:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026/07/16 02:57:31 02:55:09 INFO Pod/llmisvc-controller-manager-5d65df97bb-b9l6n: Killing - Stopping container manager 2026/07/16 02:57:31 02:55:09 INFO Pod/kserve-controller-manager-6c67dcb69c-l8cmz: Killing - Stopping container manager 2026/07/16 02:57:31 02:55:09 INFO Pod/odh-model-controller-6849fc49d8-lp8xz: Killing - Stopping container manager 2026/07/16 02:57:31 02:55:09 INFO Pod/model-serving-api-6688cd56bd-8bsfb: Killing - Stopping container server 2026/07/16 02:57:31 02:55:00 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 02:57:31 02:55:00 INFO Pod/maas-api-84449bdd5d-z55xx: Killing - Stopping container maas-api 2026/07/16 02:57:31 02:54:59 INFO Pod/maas-controller-85bdcf9988-9spvd: Killing - Stopping container manager 2026/07/16 02:57:31 02:54:29 INFO Pod/maas-api-84449bdd5d-z55xx: Started - Started container maas-api 2026/07/16 02:57:31 02:54:29 INFO Pod/maas-api-84449bdd5d-z55xx: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" in 800ms (800ms including waiting). Image size: 175018396 bytes. 2026/07/16 02:57:31 02:54:29 INFO Pod/maas-api-84449bdd5d-z55xx: Created - Created container: maas-api 2026/07/16 02:57:31 02:54:28 INFO Pod/maas-api-84449bdd5d-z55xx: Scheduled - Successfully assigned redhat-ods-applications/maas-api-84449bdd5d-z55xx to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/16 02:57:31 02:54:28 INFO Pod/maas-api-84449bdd5d-z55xx: AddedInterface - Add eth0 [10.129.3.192/23] from ovn-kubernetes 2026/07/16 02:57:31 02:54:28 INFO Pod/maas-api-84449bdd5d-z55xx: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" 2026/07/16 02:57:31 02:54:28 INFO Deployment/maas-api: ScalingReplicaSet - Scaled up replica set maas-api-84449bdd5d from 0 to 1 2026/07/16 02:57:31 02:54:28 WARN ReplicaSet/maas-api-84449bdd5d: FailedCreate - Error creating: pods "maas-api-84449bdd5d-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 2026/07/16 02:57:31 02:54:28 INFO ReplicaSet/maas-api-84449bdd5d: SuccessfulCreate - Created pod: maas-api-84449bdd5d-z55xx 2026/07/16 02:57:31 02:54:14 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-l8cmz_53d7f3ce-8a2d-438d-8943-63f63391c898 became leader 2026/07/16 02:57:31 02:54:13 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-b9l6n_176e2a64-9259-4091-9b3b-26bcc50c7613 became leader 2026/07/16 02:57:31 02:53:59 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-lp8xz_69d3ce0f-c2af-4c98-943a-6b8a718923b3 became leader 2026/07/16 02:57:31 02:53:56 INFO Pod/llmisvc-controller-manager-5d65df97bb-b9l6n: Created - Created container: manager 2026/07/16 02:57:31 02:53:56 INFO Pod/kserve-controller-manager-6c67dcb69c-l8cmz: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 1.211s (1.211s including waiting). Image size: 238435613 bytes. 2026/07/16 02:57:31 02:53:56 INFO Pod/llmisvc-controller-manager-5d65df97bb-b9l6n: Started - Started container manager 2026/07/16 02:57:31 02:53:56 INFO Pod/llmisvc-controller-manager-5d65df97bb-b9l6n: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 802ms (802ms including waiting). Image size: 236366599 bytes. 2026/07/16 02:57:31 02:53:56 INFO Lease/maas-controller.models-as-a-service.opendatahub.io: LeaderElection - maas-controller-85bdcf9988-9spvd_d0457ddd-5384-43ac-b287-a0587a02dbde became leader 2026/07/16 02:57:31 02:53:56 INFO Pod/kserve-controller-manager-6c67dcb69c-l8cmz: Started - Started container manager 2026/07/16 02:57:31 02:53:56 INFO Pod/kserve-controller-manager-6c67dcb69c-l8cmz: Created - Created container: manager 2026/07/16 02:57:31 02:53:55 INFO Pod/llmisvc-controller-manager-5d65df97bb-b9l6n: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-b9l6n to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/16 02:57:31 02:53:55 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-b9l6n 2026/07/16 02:57:31 02:53:55 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/16 02:57:31 02:53:55 INFO Pod/kserve-controller-manager-6c67dcb69c-l8cmz: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/16 02:57:31 02:53:55 INFO Pod/kserve-controller-manager-6c67dcb69c-l8cmz: AddedInterface - Add eth0 [10.131.1.215/23] from ovn-kubernetes 2026/07/16 02:57:31 02:53:55 INFO Pod/llmisvc-controller-manager-5d65df97bb-b9l6n: AddedInterface - Add eth0 [10.129.3.191/23] from ovn-kubernetes 2026/07/16 02:57:31 02:53:55 INFO Pod/llmisvc-controller-manager-5d65df97bb-b9l6n: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/16 02:57:31 02:53:54 INFO Pod/kserve-controller-manager-6c67dcb69c-l8cmz: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-l8cmz to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/16 02:57:31 02:53:54 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-l8cmz 2026/07/16 02:57:31 02:53:54 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/16 02:57:31 02:53:41 INFO Pod/model-serving-api-6688cd56bd-8bsfb: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/16 02:57:31 02:53:41 INFO Pod/model-serving-api-6688cd56bd-8bsfb: Created - Created container: server 2026/07/16 02:57:31 02:53:41 INFO Pod/model-serving-api-6688cd56bd-8bsfb: Started - Started container server 2026/07/16 02:57:31 02:53:40 INFO Pod/model-serving-api-6688cd56bd-8bsfb: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-8bsfb to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/16 02:57:31 02:53:40 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/16 02:57:31 02:53:40 INFO Pod/odh-model-controller-6849fc49d8-lp8xz: Created - Created container: manager 2026/07/16 02:57:31 02:53:40 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/16 02:57:31 02:53:40 INFO Pod/maas-controller-85bdcf9988-9spvd: Started - Started container manager 2026/07/16 02:57:31 02:53:40 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-8bsfb 2026/07/16 02:57:31 02:53:40 INFO Pod/odh-model-controller-6849fc49d8-lp8xz: Started - Started container manager 2026/07/16 02:57:31 02:53:40 INFO Pod/odh-model-controller-6849fc49d8-lp8xz: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 912ms (912ms including waiting). Image size: 238959091 bytes. 2026/07/16 02:57:31 02:53:40 INFO Pod/model-serving-api-6688cd56bd-8bsfb: AddedInterface - Add eth0 [10.131.1.214/23] from ovn-kubernetes 2026/07/16 02:57:31 02:53:40 INFO Pod/maas-controller-85bdcf9988-9spvd: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 699ms (699ms including waiting). Image size: 212866077 bytes. 2026/07/16 02:57:31 02:53:40 INFO Pod/maas-controller-85bdcf9988-9spvd: Created - Created container: manager 2026/07/16 02:57:31 02:53:39 INFO Pod/odh-model-controller-6849fc49d8-lp8xz: AddedInterface - Add eth0 [10.131.1.213/23] from ovn-kubernetes 2026/07/16 02:57:31 02:53:39 INFO Pod/maas-controller-85bdcf9988-9spvd: AddedInterface - Add eth0 [10.129.3.190/23] from ovn-kubernetes 2026/07/16 02:57:31 02:53:39 INFO Pod/maas-controller-85bdcf9988-9spvd: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/16 02:57:31 02:53:39 INFO Pod/odh-model-controller-6849fc49d8-lp8xz: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/16 02:57:31 02:53:38 INFO Pod/maas-controller-85bdcf9988-9spvd: Scheduled - Successfully assigned redhat-ods-applications/maas-controller-85bdcf9988-9spvd to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/16 02:57:31 02:53:38 INFO Pod/odh-model-controller-6849fc49d8-lp8xz: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-lp8xz to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/16 02:57:31 02:53:38 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-lp8xz 2026/07/16 02:57:31 02:53:38 WARN Pod/odh-model-controller-6849fc49d8-lp8xz: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/16 02:57:31 02:53:38 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/16 02:57:31 02:53:38 INFO Deployment/maas-controller: ScalingReplicaSet - Scaled up replica set maas-controller-85bdcf9988 from 0 to 1 2026/07/16 02:57:31 02:53:38 INFO ReplicaSet/maas-controller-85bdcf9988: SuccessfulCreate - Created pod: maas-controller-85bdcf9988-9spvd 2026/07/16 02:57:31 02:53:23 INFO Pod/maas-postgres-d66f546cf-dvhks: Pulled - Successfully pulled image "registry.redhat.io/rhel9/postgresql-15:latest" in 5.836s (5.836s including waiting). Image size: 400814167 bytes. 2026/07/16 02:57:31 02:53:23 INFO Pod/maas-postgres-d66f546cf-dvhks: Created - Created container: postgres 2026/07/16 02:57:31 02:53:23 INFO Pod/maas-postgres-d66f546cf-dvhks: Started - Started container postgres 2026/07/16 02:57:31 02:53:17 INFO Pod/maas-postgres-d66f546cf-dvhks: Pulling - Pulling image "registry.redhat.io/rhel9/postgresql-15:latest" 2026/07/16 02:57:31 02:53:17 INFO Pod/maas-postgres-d66f546cf-dvhks: AddedInterface - Add eth0 [10.129.3.189/23] from ovn-kubernetes 2026/07/16 02:57:31 02:53:16 INFO Pod/maas-postgres-d66f546cf-dvhks: Scheduled - Successfully assigned redhat-ods-applications/maas-postgres-d66f546cf-dvhks to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/16 02:57:31 02:53:16 INFO ReplicaSet/maas-postgres-d66f546cf: SuccessfulCreate - Created pod: maas-postgres-d66f546cf-dvhks 2026/07/16 02:57:31 02:53:16 INFO Deployment/maas-postgres: ScalingReplicaSet - Scaled up replica set maas-postgres-d66f546cf from 0 to 1 2026/07/16 02:57:31 02:53:07 INFO Pod/trustyai-service-operator-controller-manager-85df4c5659-gwf2m: Killing - Stopping container manager 2026/07/16 02:57:31 02:52:58 INFO Pod/maas-api-84449bdd5d-mfrt8: Killing - Stopping container maas-api 2026/07/16 02:57:31 02:52:58 INFO Lease/b7e9931f.trustyai.opendatahub.io: LeaderElection - trustyai-service-operator-controller-manager-85df4c5659-gwf2m_9652287c-182e-4ba5-b8f2-d03d2b42147a became leader 2026/07/16 02:57:31 02:52:57 INFO Pod/odh-model-controller-6849fc49d8-fdlv9: Killing - Stopping container manager 2026/07/16 02:57:31 02:52:57 INFO Pod/model-serving-api-6688cd56bd-jjdcb: Killing - Stopping container server 2026/07/16 02:57:31 02:52:57 INFO Pod/llmisvc-controller-manager-5d65df97bb-j6tx9: Killing - Stopping container manager 2026/07/16 02:57:31 02:52:57 INFO Pod/kserve-controller-manager-6c67dcb69c-f9qnp: Killing - Stopping container manager 2026/07/16 02:57:31 02:52:42 INFO Pod/trustyai-service-operator-controller-manager-85df4c5659-gwf2m: Started - Started container manager 2026/07/16 02:57:31 02:52:42 INFO Pod/trustyai-service-operator-controller-manager-85df4c5659-gwf2m: AddedInterface - Add eth0 [10.129.3.188/23] from ovn-kubernetes 2026/07/16 02:57:31 02:52:42 INFO Pod/trustyai-service-operator-controller-manager-85df4c5659-gwf2m: Pulled - Container image "registry.redhat.io/rhoai/odh-trustyai-service-operator-rhel9@sha256:17b36dfec55d1d9aeb3e778d1cf468b918e2b762efb53af1aa6af095387e4da2" already present on machine 2026/07/16 02:57:31 02:52:42 INFO Pod/trustyai-service-operator-controller-manager-85df4c5659-gwf2m: Created - Created container: manager 2026/07/16 02:57:31 02:52:41 INFO Pod/trustyai-service-operator-controller-manager-85df4c5659-gwf2m: Scheduled - Successfully assigned redhat-ods-applications/trustyai-service-operator-controller-manager-85df4c5659-gwf2m to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/16 02:57:31 02:52:41 INFO Pod/trustyai-service-operator-controller-manager-85df4c5659-6n8df: Killing - Stopping container manager 2026/07/16 02:57:31 02:52:41 INFO Pod/trustyai-service-operator-controller-manager-85df4c5659-tblbk: Killing - Stopping container manager 2026/07/16 02:57:31 02:52:41 INFO ReplicaSet/trustyai-service-operator-controller-manager-85df4c5659: SuccessfulCreate - Created pod: trustyai-service-operator-controller-manager-85df4c5659-gwf2m 2026/07/16 02:57:31 02:52:41 INFO Deployment/trustyai-service-operator-controller-manager: ScalingReplicaSet - Scaled up replica set trustyai-service-operator-controller-manager-85df4c5659 from 0 to 1 2026/07/16 02:57:31 === RESOURCE QUOTAS === 2026/07/16 02:57:31 No resource quotas found in monitored namespaces 2026/07/16 02:57:31 Classification for components: infrastructure/storage (code=1006, confidence=medium) 2026/07/16 02:57:31 Evidence: event Pod/model-serving-api-6688cd56bd-vlzxh: FailedMount - MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/model-serving-api-6688cd56bd-vlzxh: FailedMount - MountVolume.SetUp failed for volume \"tls-certs\" : secret \"model-serving-api-tls\" not found"],"confidence":"medium"} 2026/07/16 02:57:31 Diagnostics complete for failed test: components controller_test.go:484: Stopping: components test failed. 2026/07/16 02:57:31 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/16 02:57:31 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/16 02:57:32 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (1113.71s)