=== RUN TestOdhOperator/components/group_2/kueue/Validate_component_removed_to_unmanaged_transition helper_test.go:150: Setting Kueue component (default-kueue) to Removed mode to start with a clean state. helper_test.go:150: Cleaning up any existing Kueue test resources. helper_test.go:150: Cleaning up Kueue cluster-scoped resources helper_test.go:150: Will attempt to delete 2 Kueue resources helper_test.go:150: Attempting to delete ClusterQueue /default helper_test.go:150: Successfully processed deletion of ClusterQueue /default helper_test.go:150: Attempting to delete Kueue /cluster helper_test.go:150: Successfully processed deletion of Kueue /cluster helper_test.go:150: Creating test namespace (test-kueue-managed-d99nog4bmils7b1a44vg) with Kueue management annotation. helper_test.go:150: Creating Kueue ConfigMap (kueue-manager-config) to verify default Kueue resource is created correctly. helper_test.go:150: Creating Kueue ConfigMap (kueue-manager-config) in namespace (redhat-ods-applications). helper_test.go:150: Updating the management state of the component in the DataScienceCluster to Unmanaged (expecting NotReady initially due to missing operator). helper_test.go:150: Ensuring LWS Operator is installed (namespace=openshift-lws-operator, name=leader-worker-set). helper_test.go:150: Ensuring LeaderWorkerSetOperator CR exists (namespace=openshift-lws-operator, name=cluster). helper_test.go:150: Ensuring OCP Kueue Operator is installed (namespace=openshift-kueue-operator, name=kueue-operator). helper_test.go:150: Verifying the component in the DataScienceCluster transitions to Ready state. helper_test.go:150: Verifying Kueue configuration (cluster) is created with all expected frameworks. testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_2/kueue/Validate_component_removed_to_unmanaged_transition (30.69s) === RUN TestOdhOperator/components/group_2/kueue === PAUSE TestOdhOperator/components/group_2/kueue === CONT TestOdhOperator/components/group_2/kueue test_context_test.go:223: Timed out after 10.001s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:114 with: Expected resource 'cluster' of kind 'Kueue' to match condition {(["BatchJob", "Deployment", "JobSet", "LeaderWorkerSet", "MPIJob", "PaddleJob", "Pod", "PyTorchJob", "RayCluster", "RayJob", "StatefulSet", "TFJob", "TrainJob", "XGBoostJob"] - .spec.config.integrations.frameworks) | length == 0 []}. Expected <string>: map[apiVersion:kueue.openshift.io/v1 kind:Kueue metadata:map[annotations:map[platform.opendatahub.io/instance.generation:1 platform.opendatahub.io/instance.name:default-kueue platform.opendatahub.io/instance.uid:385d18d7-66de-4065-b4b1-0a0426d9fd06 platform.opendatahub.io/type:OpenShift AI Self-Managed platform.opendatahub.io/version:3.5.0-ea.2] creationTimestamp:2026-07-12T10:51:29Z deletionGracePeriodSeconds:0 deletionTimestamp:2026-07-12T11:36:43Z finalizers:[kueue.openshift.io/finalizer] generation:2 labels:map[platform.opendatahub.io/part-of:kueue] managedFields:[map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:status:map[f:conditions:map[k:{"type":"Available"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"CertManagerAvailable"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"Degraded"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"Progressing"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]]] f:readyReplicas:map[]]] manager:kueue-operator operation:Apply subresource:status time:2026-07-12T10:52:13Z] map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[.:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:labels:map[.:map[] f:platform.opendatahub.io/part-of:map[]]] f:spec:map[.:map[] f:config:map[.:map[] f:integrations:map[.:map[] f:frameworks:map[.:map[] v:"Deployment":map[] v:"Pod":map[] v:"PyTorchJob":map[] v:"RayCluster":map[] v:"RayJob":map[] v:"StatefulSet":map[] v:"TrainJob":map[]]]] f:logLevel:map[] f:managementState:map[] f:operatorLogLevel:map[]]] manager:manager operation:Update time:2026-07-12T10:51:29Z] map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:finalizers:map[.:map[] v:"kueue.openshift.io/finalizer":map[]]]] manager:kueue-operator operation:Update time:2026-07-12T11:36:43Z]] name:cluster resourceVersion:390841441 uid:909aff3e-5453-40a8-b4c7-872f72665d6f] spec:map[config:map[integrations:map[frameworks:[Deployment Pod PyTorchJob RayCluster RayJob StatefulSet TrainJob]]] logLevel:Normal managementState:Managed operatorLogLevel:Normal] status:map[conditions:[map[lastTransitionTime:2026-07-12T10:52:09Z message:All 2 replicas are ready reason:AllReplicasReady status:True type:Available] map[lastTransitionTime:2026-07-12T10:52:09Z message:Deployment is up to date reason:AsExpected status:False type:Progressing] map[lastTransitionTime:2026-07-12T10:52:09Z message: reason:AsExpected status:False type:Degraded] map[lastTransitionTime:2026-07-12T10:51:53Z message:cert-manager is installed reason:CertManagerInstalled status:True type:CertManagerAvailable]] readyReplicas:2]] to match expression <string>: (["BatchJob", "Deployment", "JobSet", "LeaderWorkerSet", "MPIJob", "PaddleJob", "Pod", "PyTorchJob", "RayCluster", "RayJob", "StatefulSet", "TFJob", "TrainJob", "XGBoostJob"] - .spec.config.integrations.frameworks) | length == 0 test_context_test.go:1106: cleanup: best-effort delete failed for /, Kind= redhat-ods-applications/kueue-manager-config: Object 'Kind' is missing in 'unstructured object has no kind' --- FAIL: TestOdhOperator/components/group_2/kueue (30.88s) === RUN TestOdhOperator/components/group_2 --- FAIL: TestOdhOperator/components/group_2 (0.00s) === RUN TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable helper_test.go:150: Verifying Tenant models-as-a-service/default-tenant is present before disable helper_test.go:150: Disabling MaaS subcomponent (setting to Removed) helper_test.go:150: Waiting for Tenant models-as-a-service/default-tenant to be deleted testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable (430.17s) === RUN TestOdhOperator/components/group_4/modelsasservice === PAUSE TestOdhOperator/components/group_4/modelsasservice === CONT TestOdhOperator/components/group_4/modelsasservice 2026-07-12T11:44:32Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.19.18"}} controller_test.go:753: Creating MaaS PostgreSQL instance in namespace redhat-ods-applications controller_test.go:753: MaaS PostgreSQL instance and maas-db-config secret created in namespace redhat-ods-applications controller_test.go:753: Creating MaaS GatewayClass and Gateway: openshift-ingress/maas-default-gateway controller_test.go:753: Using hostname=maas.apps.ods-qe-psi-07.osp.rh-ods.com, certSecret=router-certs-default controller_test.go:753: MaaS Gateway openshift-ingress/maas-default-gateway created with HTTP+HTTPS listeners test_context_test.go:615: Timed out after 420.007s. The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1824 with: Expected resource list 'models-as-a-service/default-tenant' of kind 'Tenant' to be empty, but it contains resources. Expected <[]unstructured.Unstructured | len:1, cap:1>: [ { Object: { "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "kind": <string>"Tenant", "metadata": <map[string]interface {} | len:8>{ "namespace": <string>"models-as-a-service", "ownerReferences": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:4>{ "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "kind": <string>"Config", "name": <string>"default", "uid": <string>"c221c951-7ace-4064-b9f4-72de27f65e08", }, ], "resourceVersion": <string>"390852542", "uid": <string>"bc338efa-9758-4ba0-b9be-9663b9cfefca", "creationTimestamp": <string>"2026-07-12T11:45:32Z", "generation": <int64>1, "managedFields": <[]interface {} | len:2, cap:2>[ <map[string]interface {} | len:6>{ "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:2>{ "f:spec": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "f:gatewayRef": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:name": <map[string]interface {} | len:0>{}, "f:namespace": <map[string]interface {} | len:0>{}, }, }, "f:metadata": <map[string]interface {} | len:1>{ "f:ownerReferences": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "k:{\"uid\":\"c221c951-7ace-4064-b9f4-72de27f65e08\"}": <map[string]interface {} | len:0>{}, }, }, }, "manager": <string>"manager", "operation": <string>"Update", "time": <string>"2026-07-12T11:45:32Z", }, <map[string]interface {} | len:7>{ "operation": <string>"Update", "subresource": <string>"status", "time": <string>"2026-07-12T11:45:55Z", "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:1>{ "f:status": <map[string]interface {} | len:3>{ "f:conditions": <map[string]interface {} | len:0>{}, "f:phase": <map[string]interface {} | len:0>{}, ".": <map[string]interface {} | len:0>{}, }, }, "manager": <string>"manager", }, ], "name": <string>"default-tenant", }, "spec": <map[string]interface {} | len:1>{ "gatewayRef": <map[string]interface {} | len:2>{ "name": <string>"maas-default-gateway", "namespace": <string>"openshift-ingress", }, }, "status": <map[string]interface {} | len:2>{ "conditions": <[]interface {} | len:5, cap:8>[ <map[string]interface {} | len:6>{ "observedGeneration": <int64>1, "reason": <string>"DependenciesMet", "status": <string>"True", "type": <string>"DependenciesAvailable", "lastTransitionTime": <string>"2026-07-12T11:45:34Z", "message": <string>"AuthConfig CRD (Kuadrant) is available on the cluster", }, <map[string]interface {} | len:6>{ "observedGeneration": <int64>1, "reason": <string>"PrerequisitesMet", "status": <string>"True", "type": <string>"MaaSPrerequisitesAvailable", "lastTransitionTime": <string>"2026-07-12T11:45:34Z", "message": <string>"All prerequisites are satisfied", }, <map[string]interface {} | len:6>{ "reason": <string>"PrerequisitesMet", "status": <string>"False", "type": <string>"Degraded", "lastTransitionTime": <string>"2026-07-12T11:45:34Z", "message": <string>"All prerequisites are satisfied", "observedGeneration": <int64>1, }, <map[string]interface {} | len:6>{ "message": <string>"maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"DeploymentsReady", "status": <string>"True", "type": <string>"DeploymentsAvailable", "lastTransitionTime": <string>"2026-07-12T11:45:55Z", }, <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-07-12T11:45:55Z", "message": <string>"MaaS platform manifests applied and maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"Reconciled", "status": <string>"True", "type": <string>"Ready", }, ], "phase": <string>"Active", }, }, }, ] to be empty --- FAIL: TestOdhOperator/components/group_4/modelsasservice (521.25s) === RUN TestOdhOperator/components/group_4 --- FAIL: TestOdhOperator/components/group_4 (0.00s) === RUN TestOdhOperator/components 2026/07/12 11:40:24 TEST FAILURE DETECTED: group 2 2026/07/12 11:40:24 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 2) === 2026/07/12 11:40:24 === CLUSTER STATE === 2026/07/12 11:40:24 Node: ods-qe-psi-07-b4mqp-master-0 2026/07/12 11:40:24 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:40:24 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:40:24 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:40:24 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:40:24 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/12 11:40:24 Node: ods-qe-psi-07-b4mqp-master-1 2026/07/12 11:40:24 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:40:24 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:40:24 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:40:24 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:40:24 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/12 11:40:24 Node: ods-qe-psi-07-b4mqp-master-2 2026/07/12 11:40:24 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:40:24 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:40:24 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:40:24 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:40:24 Resources: 11500m CPU, 23450732Ki memory allocatable (of 12000m CPU, 24601708Ki memory capacity) 2026/07/12 11:40:24 Node: ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/12 11:40:24 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:40:24 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:40:24 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:40:24 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:40:24 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/12 11:40:24 Node: ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/12 11:40:24 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:40:24 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:40:24 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:40:24 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:40:24 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/12 11:40:24 Node: ods-qe-psi-07-b4mqp-worker-0-jd4pm 2026/07/12 11:40:24 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:40:24 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:40:24 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:40:24 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:40:24 Resources: 11500m CPU, 23452736Ki memory allocatable (of 12000m CPU, 24603712Ki memory capacity) 2026/07/12 11:40:24 Node: ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/12 11:40:24 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:40:24 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:40:24 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:40:24 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:40:24 Resources: 11500m CPU, 23452744Ki memory allocatable (of 12000m CPU, 24603720Ki memory capacity) 2026/07/12 11:40:24 Node: ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/12 11:40:24 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:40:24 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:40:24 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:40:24 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:40:24 Resources: 11500m CPU, 23450720Ki memory allocatable (of 12000m CPU, 24601696Ki memory capacity) 2026/07/12 11:40:24 === DEPLOYMENTS === 2026/07/12 11:40:24 Namespace: redhat-ods-applications 2026/07/12 11:40:24 Deployment maas-api: 1/1 (READY) 2026/07/12 11:40:24 Deployment maas-controller: 1/1 (READY) 2026/07/12 11:40:24 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/12 11:40:24 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/12 11:40:24 Namespace: kube-system 2026/07/12 11:40:24 === PODS === 2026/07/12 11:40:24 Namespace redhat-ods-applications: no problematic pods found 2026/07/12 11:40:24 Namespace kube-system: no problematic pods found 2026/07/12 11:40:24 === OPERATOR STATUS === 2026/07/12 11:40:24 Operator deployment: 3/3 ready 2026/07/12 11:40:24 === DSCI STATUS === 2026/07/12 11:40:24 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/12 11:40:24 === DSC STATUS === 2026/07/12 11:40:24 Failed to collect DSC data: default-dsc unhealthy conditions: ModelsAsServiceReady=False 2026/07/12 11:40:24 === RECENT EVENTS === 2026/07/12 11:40:24 11:40:16 INFO Pod/odh-model-controller-867855946f-b5grz: Killing - Stopping container manager 2026/07/12 11:40:24 11:40:16 INFO Pod/model-registry-operator-controller-manager-76b5b7f674-gfnjz: Killing - Stopping container manager 2026/07/12 11:40:24 11:40:16 INFO Pod/model-serving-api-6688cd56bd-xc2pr: Killing - Stopping container server 2026/07/12 11:40:24 11:40:16 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-55cjv: Killing - Stopping container manager 2026/07/12 11:40:24 11:40:15 INFO Pod/kserve-controller-manager-6c67dcb69c-zs6tq: Killing - Stopping container manager 2026/07/12 11:40:24 11:40:15 INFO Pod/llmisvc-controller-manager-5d65df97bb-jd74j: Killing - Stopping container manager 2026/07/12 11:40:24 11:40:02 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-867855946f-b5grz_b3926d03-3ed9-431f-9df5-228b0e9e09e9 became leader 2026/07/12 11:40:24 11:40:00 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-5bd688895b-55cjv_25fa3bb9-a00e-4ec1-be53-8902bd0e7332 became leader 2026/07/12 11:40:24 11:40:00 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-55cjv: AddedInterface - Add eth0 [10.131.3.47/23] from ovn-kubernetes 2026/07/12 11:40:24 11:40:00 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-55cjv: Pulled - Container image "registry.redhat.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" already present on machine 2026/07/12 11:40:24 11:40:00 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-55cjv: Created - Created container: manager 2026/07/12 11:40:24 11:40:00 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-55cjv: Started - Started container manager 2026/07/12 11:40:24 11:39:59 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-55cjv: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-5bd688895b-55cjv to ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/12 11:40:24 11:39:59 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-5bd688895b-55cjv 2026/07/12 11:40:24 11:39:59 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-5bd688895b from 0 to 1 2026/07/12 11:40:24 11:39:59 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-55nn6: Killing - Stopping container manager 2026/07/12 11:40:24 11:39:59 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-td5pc: Killing - Stopping container manager 2026/07/12 11:40:24 11:39:45 INFO Pod/odh-model-controller-867855946f-b5grz: AddedInterface - Add eth0 [10.131.3.46/23] from ovn-kubernetes 2026/07/12 11:40:24 11:39:45 INFO Pod/odh-model-controller-867855946f-b5grz: Started - Started container manager 2026/07/12 11:40:24 11:39:45 INFO Pod/odh-model-controller-867855946f-b5grz: Created - Created container: manager 2026/07/12 11:40:24 11:39:45 INFO Pod/odh-model-controller-867855946f-b5grz: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 807ms (807ms including waiting). Image size: 238959091 bytes. 2026/07/12 11:40:24 11:39:45 INFO Pod/odh-model-controller-867855946f-b5grz: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/12 11:40:24 11:39:44 INFO Pod/odh-model-controller-867855946f-b5grz: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-867855946f-b5grz to ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/12 11:40:24 11:39:44 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-867855946f from 0 to 1 2026/07/12 11:40:24 11:39:44 INFO Pod/odh-model-controller-867855946f-dsf99: Killing - Stopping container manager 2026/07/12 11:40:24 11:39:44 INFO Pod/odh-model-controller-867855946f-qqz9l: Killing - Stopping container manager 2026/07/12 11:40:24 11:39:44 INFO ReplicaSet/odh-model-controller-867855946f: SuccessfulCreate - Created pod: odh-model-controller-867855946f-b5grz 2026/07/12 11:40:24 11:39:30 INFO Pod/model-serving-api-6688cd56bd-xc2pr: Created - Created container: server 2026/07/12 11:40:24 11:39:30 WARN Pod/model-serving-api-6688cd56bd-8csdg: Unhealthy - Readiness probe failed: Get "https://10.130.2.128:8443/readyz": context deadline exceeded 2026/07/12 11:40:24 11:39:30 INFO Pod/model-serving-api-6688cd56bd-xc2pr: Started - Started container server 2026/07/12 11:40:24 11:39:29 INFO Pod/model-serving-api-6688cd56bd-xc2pr: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-xc2pr to ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/12 11:40:24 11:39:29 INFO Pod/model-serving-api-6688cd56bd-xc2pr: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/12 11:40:24 11:39:29 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/12 11:40:24 11:39:29 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-xc2pr 2026/07/12 11:40:24 11:39:29 INFO Pod/model-serving-api-6688cd56bd-6g7mh: Killing - Stopping container server 2026/07/12 11:40:24 11:39:29 INFO Pod/model-serving-api-6688cd56bd-8csdg: Killing - Stopping container server 2026/07/12 11:40:24 11:39:29 INFO Pod/model-serving-api-6688cd56bd-xc2pr: AddedInterface - Add eth0 [10.131.3.45/23] from ovn-kubernetes 2026/07/12 11:40:24 11:38:31 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-55nn6: Created - Created container: manager 2026/07/12 11:40:24 11:38:31 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-55nn6: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" in 1.932s (1.932s including waiting). Image size: 195240093 bytes. 2026/07/12 11:40:24 11:38:31 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-55nn6: Started - Started container manager 2026/07/12 11:40:24 11:38:29 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-55nn6: Pulling - Pulling image "registry.redhat.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" 2026/07/12 11:40:24 11:38:29 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-55nn6: AddedInterface - Add eth0 [10.131.0.218/23] from ovn-kubernetes 2026/07/12 11:40:24 11:38:28 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-55nn6: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-5bd688895b-55nn6 to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/12 11:40:24 11:38:28 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-5bd688895b from 1 to 2 2026/07/12 11:40:24 11:38:28 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-5bd688895b-55nn6 2026/07/12 11:40:24 11:38:09 INFO Pod/odh-model-controller-867855946f-qqz9l: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 664ms (664ms including waiting). Image size: 238959091 bytes. 2026/07/12 11:40:24 11:38:09 INFO Pod/odh-model-controller-867855946f-qqz9l: Created - Created container: manager 2026/07/12 11:40:24 11:38:09 INFO Pod/odh-model-controller-867855946f-qqz9l: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/12 11:40:24 11:38:09 INFO Pod/odh-model-controller-867855946f-qqz9l: Started - Started container manager 2026/07/12 11:40:24 11:38:09 INFO Pod/odh-model-controller-867855946f-qqz9l: AddedInterface - Add eth0 [10.131.3.44/23] from ovn-kubernetes 2026/07/12 11:40:24 11:38:08 INFO Pod/odh-model-controller-867855946f-qqz9l: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-867855946f-qqz9l to ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/12 11:40:24 11:38:08 INFO ReplicaSet/odh-model-controller-867855946f: SuccessfulCreate - Created pod: odh-model-controller-867855946f-qqz9l 2026/07/12 11:40:24 11:38:08 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-867855946f from 1 to 2 2026/07/12 11:40:24 11:37:49 INFO Pod/model-serving-api-6688cd56bd-6g7mh: AddedInterface - Add eth0 [10.131.3.43/23] from ovn-kubernetes 2026/07/12 11:40:24 11:37:49 INFO Pod/model-serving-api-6688cd56bd-6g7mh: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/12 11:40:24 11:37:49 INFO Pod/model-serving-api-6688cd56bd-6g7mh: Created - Created container: server 2026/07/12 11:40:24 11:37:49 INFO Pod/model-serving-api-6688cd56bd-6g7mh: Started - Started container server 2026/07/12 11:40:24 11:37:48 INFO Pod/model-serving-api-6688cd56bd-6g7mh: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-6g7mh to ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/12 11:40:24 11:37:48 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-6g7mh 2026/07/12 11:40:24 11:37:48 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 1 to 2 2026/07/12 11:40:24 11:37:26 INFO Pod/maas-api-84449bdd5d-s4gb7: Started - Started container maas-api 2026/07/12 11:40:24 11:37:26 INFO Pod/maas-api-84449bdd5d-s4gb7: Created - Created container: maas-api 2026/07/12 11:40:24 11:37:26 INFO Pod/maas-api-84449bdd5d-s4gb7: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" in 1.551s (1.551s including waiting). Image size: 175018396 bytes. 2026/07/12 11:40:24 11:37:24 INFO Pod/maas-api-84449bdd5d-s4gb7: Scheduled - Successfully assigned redhat-ods-applications/maas-api-84449bdd5d-s4gb7 to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/12 11:40:24 11:37:24 INFO Pod/maas-api-84449bdd5d-s4gb7: AddedInterface - Add eth0 [10.131.0.215/23] from ovn-kubernetes 2026/07/12 11:40:24 11:37:24 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/12 11:40:24 11:37:24 INFO Pod/maas-api-84449bdd5d-s4gb7: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" 2026/07/12 11:40:24 11:37:24 WARN ReplicaSet/maas-api-84449bdd5d: FailedCreate - Error creating: pods "maas-api-84449bdd5d-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 2026/07/12 11:40:24 11:37:24 INFO ReplicaSet/maas-api-84449bdd5d: SuccessfulCreate - Created pod: maas-api-84449bdd5d-s4gb7 2026/07/12 11:40:24 11:37:23 INFO Deployment/maas-api: ScalingReplicaSet - Scaled up replica set maas-api-84449bdd5d from 0 to 1 2026/07/12 11:40:24 11:37:11 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-jd74j_ac9383ea-ced1-4a68-891c-595b3bf4f787 became leader 2026/07/12 11:40:24 11:37:08 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-zs6tq_42d7e1ac-31ff-4de0-9865-00b1281f673d became leader 2026/07/12 11:40:24 11:37:02 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-td5pc: Created - Created container: manager 2026/07/12 11:40:24 11:37:02 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-5bd688895b-td5pc_0ee05b44-f2f9-4f25-9e26-f8ee29d8a854 became leader 2026/07/12 11:40:24 11:37:02 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-td5pc: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" in 2.845s (2.845s including waiting). Image size: 195240093 bytes. 2026/07/12 11:40:24 11:37:02 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-td5pc: Started - Started container manager 2026/07/12 11:40:24 11:36:59 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-td5pc: Pulling - Pulling image "registry.redhat.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" 2026/07/12 11:40:24 11:36:59 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-td5pc: AddedInterface - Add eth0 [10.131.3.41/23] from ovn-kubernetes 2026/07/12 11:40:24 11:36:58 WARN Pod/workload-variant-autoscaler-controller-manager-5bd688895b-td5pc: FailedMount - MountVolume.SetUp failed for volume "epp-metrics-token" : secret "workload-variant-autoscaler-epp-metrics-token" not found 2026/07/12 11:40:24 11:36:58 WARN ServiceMonitor/workload-variant-autoscaler-controller-manager-metrics-monitor: InvalidConfiguration - ServiceMonitor workload-variant-autoscaler-controller-manager-metrics-monitor was rejected due to invalid configuration: unable to get secret "workload-variant-autoscaler-controller-manager-token": secrets "workload-variant-autoscaler-controller-manager-token" not found 2026/07/12 11:40:24 11:36:57 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-td5pc: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-5bd688895b-td5pc to ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/12 11:40:24 11:36:57 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-5bd688895b-td5pc 2026/07/12 11:40:24 11:36:57 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-5bd688895b from 0 to 1 2026/07/12 11:40:24 11:36:57 WARN ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: FailedCreate - Error creating: pods "workload-variant-autoscaler-controller-manager-5bd688895b-" is forbidden: error looking up service account redhat-ods-applications/workload-variant-autoscaler-controller-manager: serviceaccount "workload-variant-autoscaler-controller-manager" not found 2026/07/12 11:40:24 11:36:54 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - ServiceMonitor controller-manager-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/12 11:40:24 11:36:54 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-867855946f-dsf99_ba86b31c-1711-4f94-ab53-72b14ed997df became leader 2026/07/12 11:40:24 11:36:52 INFO Lease/85f368d1.opendatahub.io: LeaderElection - model-registry-operator-controller-manager-76b5b7f674-gfnjz_d406e8b8-3331-4785-b21f-bf72dbe281dc became leader 2026/07/12 11:40:24 11:36:52 INFO Pod/kserve-controller-manager-6c67dcb69c-zs6tq: Started - Started container manager 2026/07/12 11:40:24 11:36:52 INFO Pod/kserve-controller-manager-6c67dcb69c-zs6tq: Created - Created container: manager 2026/07/12 11:40:24 11:36:52 INFO Pod/kserve-controller-manager-6c67dcb69c-zs6tq: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 2.083s (2.083s including waiting). Image size: 238435613 bytes. 2026/07/12 11:40:24 11:36:52 INFO Lease/maas-controller.models-as-a-service.opendatahub.io: LeaderElection - maas-controller-85bdcf9988-6vrrn_0d39ee67-36d0-4de1-9ee6-d754f785c0ea became leader 2026/07/12 11:40:24 11:36:51 INFO Pod/llmisvc-controller-manager-5d65df97bb-jd74j: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 731ms (731ms including waiting). Image size: 236366599 bytes. 2026/07/12 11:40:24 11:36:51 INFO Pod/llmisvc-controller-manager-5d65df97bb-jd74j: Created - Created container: manager 2026/07/12 11:40:24 11:36:51 INFO Pod/llmisvc-controller-manager-5d65df97bb-jd74j: Started - Started container manager 2026/07/12 11:40:24 11:36:50 INFO Pod/llmisvc-controller-manager-5d65df97bb-jd74j: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-jd74j to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/12 11:40:24 11:36:50 INFO Pod/kserve-controller-manager-6c67dcb69c-zs6tq: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-zs6tq to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/12 11:40:24 11:36:50 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-jd74j 2026/07/12 11:40:24 11:36:50 INFO Pod/kserve-controller-manager-6c67dcb69c-zs6tq: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/12 11:40:24 11:36:50 INFO Pod/kserve-controller-manager-6c67dcb69c-zs6tq: AddedInterface - Add eth0 [10.130.2.129/23] from ovn-kubernetes 2026/07/12 11:40:24 11:36:50 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-zs6tq 2026/07/12 11:40:24 11:36:50 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/12 11:40:24 11:36:50 INFO Pod/llmisvc-controller-manager-5d65df97bb-jd74j: AddedInterface - Add eth0 [10.129.2.92/23] from ovn-kubernetes 2026/07/12 11:40:24 11:36:50 INFO Pod/llmisvc-controller-manager-5d65df97bb-jd74j: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/12 11:40:24 11:36:50 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/12 11:40:24 11:36:37 INFO Pod/model-serving-api-6688cd56bd-8csdg: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/12 11:40:24 11:36:37 INFO Pod/model-serving-api-6688cd56bd-8csdg: AddedInterface - Add eth0 [10.130.2.128/23] from ovn-kubernetes 2026/07/12 11:40:24 11:36:37 INFO Pod/odh-model-controller-867855946f-dsf99: Created - Created container: manager 2026/07/12 11:40:24 11:36:37 INFO Pod/odh-model-controller-867855946f-dsf99: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 1.049s (1.049s including waiting). Image size: 238959091 bytes. 2026/07/12 11:40:24 11:36:37 INFO Pod/odh-model-controller-867855946f-dsf99: Started - Started container manager 2026/07/12 11:40:24 11:36:37 INFO Pod/maas-controller-85bdcf9988-6vrrn: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 2.073s (2.073s including waiting). Image size: 212866077 bytes. 2026/07/12 11:40:24 11:36:37 INFO Pod/maas-controller-85bdcf9988-6vrrn: Created - Created container: manager 2026/07/12 11:40:24 11:36:37 INFO Pod/maas-controller-85bdcf9988-6vrrn: Started - Started container manager 2026/07/12 11:40:24 11:36:37 INFO Pod/model-serving-api-6688cd56bd-8csdg: Started - Started container server 2026/07/12 11:40:24 11:36:37 INFO Pod/model-serving-api-6688cd56bd-8csdg: Created - Created container: server 2026/07/12 11:40:24 11:36:36 INFO Pod/model-serving-api-6688cd56bd-8csdg: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-8csdg to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/12 11:40:24 11:36:36 INFO Pod/odh-model-controller-867855946f-dsf99: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/12 11:40:24 11:36:36 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/12 11:40:24 11:36:36 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-8csdg 2026/07/12 11:40:24 11:36:36 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/12 11:40:24 11:36:36 INFO Pod/odh-model-controller-867855946f-dsf99: AddedInterface - Add eth0 [10.130.2.127/23] from ovn-kubernetes 2026/07/12 11:40:24 11:36:35 INFO Pod/model-registry-operator-controller-manager-76b5b7f674-gfnjz: AddedInterface - Add eth0 [10.130.2.125/23] from ovn-kubernetes 2026/07/12 11:40:24 11:36:35 INFO Pod/model-registry-operator-controller-manager-76b5b7f674-gfnjz: Started - Started container manager 2026/07/12 11:40:24 11:36:35 INFO Pod/maas-controller-85bdcf9988-6vrrn: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/12 11:40:24 11:36:35 INFO Pod/maas-controller-85bdcf9988-6vrrn: AddedInterface - Add eth0 [10.130.2.126/23] from ovn-kubernetes 2026/07/12 11:40:24 11:36:35 WARN Pod/odh-model-controller-867855946f-dsf99: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/12 11:40:24 11:36:35 INFO Pod/model-registry-operator-controller-manager-76b5b7f674-gfnjz: Pulled - Container image "registry.redhat.io/rhoai/odh-model-registry-operator-rhel9@sha256:ebcec5c43468283ed3cbab50530f939166f8f3e8c9d306e91a8107d05d7fa9d4" already present on machine 2026/07/12 11:40:24 11:36:35 INFO Pod/model-registry-operator-controller-manager-76b5b7f674-gfnjz: Created - Created container: manager 2026/07/12 11:40:24 11:36:34 INFO Pod/odh-model-controller-867855946f-dsf99: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-867855946f-dsf99 to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/12 11:40:24 11:36:34 INFO Pod/maas-controller-85bdcf9988-6vrrn: Scheduled - Successfully assigned redhat-ods-applications/maas-controller-85bdcf9988-6vrrn to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/12 11:40:24 11:36:34 INFO Pod/model-registry-operator-controller-manager-76b5b7f674-gfnjz: Scheduled - Successfully assigned redhat-ods-applications/model-registry-operator-controller-manager-76b5b7f674-gfnjz to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/12 11:40:24 11:36:34 WARN Pod/model-registry-operator-controller-manager-76b5b7f674-gfnjz: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 2026/07/12 11:40:24 11:36:34 INFO Deployment/model-registry-operator-controller-manager: ScalingReplicaSet - Scaled up replica set model-registry-operator-controller-manager-76b5b7f674 from 0 to 1 2026/07/12 11:40:24 11:36:34 INFO ReplicaSet/odh-model-controller-867855946f: SuccessfulCreate - Created pod: odh-model-controller-867855946f-dsf99 2026/07/12 11:40:24 11:36:34 INFO ReplicaSet/maas-controller-85bdcf9988: SuccessfulCreate - Created pod: maas-controller-85bdcf9988-6vrrn 2026/07/12 11:40:24 11:36:34 INFO ReplicaSet/model-registry-operator-controller-manager-76b5b7f674: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-76b5b7f674-gfnjz 2026/07/12 11:40:24 11:36:34 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-867855946f from 0 to 1 2026/07/12 11:40:24 11:36:34 INFO Deployment/maas-controller: ScalingReplicaSet - Scaled up replica set maas-controller-85bdcf9988 from 0 to 1 2026/07/12 11:40:24 11:36:26 INFO Deployment/rhods-dashboard: ScalingReplicaSet - Scaled up replica set rhods-dashboard-d8f68f65 from 0 to 2 2026/07/12 11:40:24 11:36:26 WARN ReplicaSet/rhods-dashboard-d8f68f65: FailedCreate - Error creating: pods "rhods-dashboard-d8f68f65-" is forbidden: error looking up service account redhat-ods-applications/rhods-dashboard: serviceaccount "rhods-dashboard" not found 2026/07/12 11:40:24 11:36:24 INFO Pod/dashboard-redirect-5bd755c985-g6z75: Killing - Stopping container nginx 2026/07/12 11:40:24 11:36:24 INFO Pod/dashboard-redirect-5bd755c985-lr2tn: Killing - Stopping container nginx 2026/07/12 11:40:24 11:36:23 WARN Pod/rhods-dashboard-d8f68f65-8n8vq: Unhealthy - Readiness probe failed: Get "https://10.129.2.91:8444/healthz": dial tcp 10.129.2.91:8444: connect: connection refused 2026/07/12 11:40:24 11:36:23 WARN Endpoints/rhods-dashboard: FailedToUpdateEndpoint - Failed to update endpoint redhat-ods-applications/rhods-dashboard: Operation cannot be fulfilled on endpoints "rhods-dashboard": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/rhods-dashboard, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: f0b82326-d555-4614-9c8e-2555e8523c3e, UID in object meta: 2026/07/12 11:40:24 11:35:57 INFO Pod/spark-operator-controller-767899d44f-mtx65: Killing - Stopping container controller 2026/07/12 11:40:24 11:35:57 INFO Pod/spark-operator-webhook-9cb9bdf44-5cfq6: Killing - Stopping container webhook 2026/07/12 11:40:24 11:35:44 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" in 1.522s (1.522s including waiting). Image size: 185092464 bytes. 2026/07/12 11:40:24 11:35:43 INFO Lease/spark-operator-controller-lock: LeaderElection - spark-operator-controller-767899d44f-mtx65_15abb371-fc34-44f8-91f9-3e1fc3831c69 became leader 2026/07/12 11:40:24 11:35:42 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: Started - Started container mlflow-ui 2026/07/12 11:40:24 11:35:42 INFO Pod/spark-operator-webhook-9cb9bdf44-5cfq6: Created - Created container: webhook 2026/07/12 11:40:24 11:35:42 INFO Pod/spark-operator-webhook-9cb9bdf44-5cfq6: Started - Started container webhook 2026/07/12 11:40:24 11:35:42 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: Started - Started container maas-ui 2026/07/12 11:40:24 11:35:42 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" 2026/07/12 11:40:24 11:35:42 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" 2026/07/12 11:40:24 11:35:42 INFO Pod/spark-operator-webhook-9cb9bdf44-5cfq6: AddedInterface - Add eth0 [10.130.2.124/23] from ovn-kubernetes 2026/07/12 11:40:24 11:35:42 INFO Pod/spark-operator-webhook-9cb9bdf44-5cfq6: Pulled - Container image "registry.redhat.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/12 11:40:24 11:35:42 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: Created - Created container: mlflow-ui 2026/07/12 11:40:24 11:35:42 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" in 565ms (565ms including waiting). Image size: 177808591 bytes. 2026/07/12 11:40:24 11:35:41 INFO Pod/spark-operator-webhook-9cb9bdf44-5cfq6: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-webhook-9cb9bdf44-5cfq6 to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/12 11:40:24 11:35:41 INFO Pod/spark-operator-webhook-9cb9bdf44-fxh4z: Killing - Stopping container webhook 2026/07/12 11:40:24 11:35:41 INFO Deployment/spark-operator-webhook: ScalingReplicaSet - Scaled up replica set spark-operator-webhook-9cb9bdf44 from 0 to 1 2026/07/12 11:40:24 11:35:41 INFO Pod/spark-operator-webhook-9cb9bdf44-mnwwz: Killing - Stopping container webhook 2026/07/12 11:40:24 11:35:41 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" in 1.55s (1.55s including waiting). Image size: 187802205 bytes. 2026/07/12 11:40:24 11:35:41 INFO ReplicaSet/spark-operator-webhook-9cb9bdf44: SuccessfulCreate - Created pod: spark-operator-webhook-9cb9bdf44-5cfq6 2026/07/12 11:40:24 11:35:41 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: Created - Created container: maas-ui 2026/07/12 11:40:24 11:35:40 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" 2026/07/12 11:40:24 11:35:40 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" in 2.301s (2.301s including waiting). Image size: 241242949 bytes. 2026/07/12 11:40:24 11:35:40 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: Created - Created container: gen-ai-ui 2026/07/12 11:40:24 11:35:40 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: Started - Started container gen-ai-ui 2026/07/12 11:40:24 11:35:37 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: Started - Started container model-registry-ui 2026/07/12 11:40:24 11:35:37 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" in 1.72s (1.721s including waiting). Image size: 182438697 bytes. 2026/07/12 11:40:24 11:35:37 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: Created - Created container: model-registry-ui 2026/07/12 11:40:24 11:35:37 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" 2026/07/12 11:40:24 11:35:36 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: Created - Created container: kube-rbac-proxy 2026/07/12 11:40:24 11:35:36 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: Started - Started container kube-rbac-proxy 2026/07/12 11:40:24 11:35:36 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" 2026/07/12 11:40:24 11:35:36 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: Created - Created container: rhods-dashboard 2026/07/12 11:40:24 11:35:36 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: Started - Started container rhods-dashboard 2026/07/12 11:40:24 11:35:36 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: Pulled - Container image "registry.redhat.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:1e614e46799fd020ac18d95c06a463421154ce86e59ccb3fa6007e66fed9910e" already present on machine 2026/07/12 11:40:24 11:35:35 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" in 7.262s (7.262s including waiting). Image size: 799822033 bytes. 2026/07/12 11:40:24 11:35:33 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" in 638ms (638ms including waiting). Image size: 185092464 bytes. 2026/07/12 11:40:24 11:35:32 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: Created - Created container: mlflow-ui 2026/07/12 11:40:24 11:35:32 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" 2026/07/12 11:40:24 11:35:32 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: Started - Started container mlflow-ui 2026/07/12 11:40:24 11:35:32 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" in 675ms (675ms including waiting). Image size: 177808591 bytes. 2026/07/12 11:40:24 11:35:32 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" 2026/07/12 11:40:24 11:35:32 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: Started - Started container maas-ui 2026/07/12 11:40:24 11:35:32 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: Created - Created container: maas-ui 2026/07/12 11:40:24 11:35:32 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" in 817ms (817ms including waiting). Image size: 187802205 bytes. 2026/07/12 11:40:24 11:35:31 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" 2026/07/12 11:40:24 11:35:31 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: Started - Started container gen-ai-ui 2026/07/12 11:40:24 11:35:31 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: Created - Created container: gen-ai-ui 2026/07/12 11:40:24 11:35:31 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" in 831ms (831ms including waiting). Image size: 241242949 bytes. 2026/07/12 11:40:24 11:35:30 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" 2026/07/12 11:40:24 11:35:30 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: Started - Started container model-registry-ui 2026/07/12 11:40:24 11:35:30 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: Created - Created container: model-registry-ui 2026/07/12 11:40:24 11:35:30 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" in 631ms (631ms including waiting). Image size: 182438697 bytes. 2026/07/12 11:40:24 11:35:29 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: Pulling - Pulling image "registry.redhat.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" 2026/07/12 11:40:24 11:35:29 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: Started - Started container kube-rbac-proxy 2026/07/12 11:40:24 11:35:29 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: Created - Created container: kube-rbac-proxy 2026/07/12 11:40:24 11:35:29 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: Pulled - Container image "registry.redhat.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:1e614e46799fd020ac18d95c06a463421154ce86e59ccb3fa6007e66fed9910e" already present on machine 2026/07/12 11:40:24 11:35:29 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: Started - Started container rhods-dashboard 2026/07/12 11:40:24 11:35:29 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: Created - Created container: rhods-dashboard 2026/07/12 11:40:24 11:35:29 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" in 820ms (820ms including waiting). Image size: 799822033 bytes. 2026/07/12 11:40:24 11:35:28 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: AddedInterface - Add eth0 [10.129.2.91/23] from ovn-kubernetes 2026/07/12 11:40:24 11:35:28 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: Pulling - Pulling image "registry.redhat.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" 2026/07/12 11:40:24 11:35:28 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: Pulling - Pulling image "registry.redhat.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" 2026/07/12 11:40:24 11:35:28 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: AddedInterface - Add eth0 [10.131.0.212/23] from ovn-kubernetes 2026/07/12 11:40:24 11:35:27 INFO Pod/rhods-dashboard-d8f68f65-8n8vq: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-d8f68f65-8n8vq to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/12 11:40:24 11:35:27 INFO Pod/rhods-dashboard-d8f68f65-bq6nx: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-d8f68f65-bq6nx to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/12 11:40:24 11:35:27 INFO Pod/spark-operator-controller-767899d44f-mtx65: Pulled - Container image "registry.redhat.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/12 11:40:24 11:35:27 INFO Deployment/rhods-dashboard: ScalingReplicaSet - Scaled up replica set rhods-dashboard-d8f68f65 from 0 to 2 2026/07/12 11:40:24 11:35:27 INFO ReplicaSet/rhods-dashboard-d8f68f65: SuccessfulCreate - Created pod: rhods-dashboard-d8f68f65-8n8vq 2026/07/12 11:40:24 11:35:27 INFO ReplicaSet/rhods-dashboard-d8f68f65: SuccessfulCreate - Created pod: rhods-dashboard-d8f68f65-bq6nx 2026/07/12 11:40:24 11:35:27 INFO Pod/spark-operator-controller-767899d44f-mtx65: AddedInterface - Add eth0 [10.130.2.123/23] from ovn-kubernetes 2026/07/12 11:40:24 11:35:27 INFO Pod/spark-operator-controller-767899d44f-mtx65: Created - Created container: controller 2026/07/12 11:40:24 11:35:27 INFO Pod/spark-operator-controller-767899d44f-mtx65: Started - Started container controller 2026/07/12 11:40:24 11:35:26 INFO Pod/spark-operator-controller-767899d44f-mtx65: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-controller-767899d44f-mtx65 to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/12 11:40:24 11:35:26 INFO Pod/spark-operator-controller-767899d44f-ml4wc: Killing - Stopping container controller 2026/07/12 11:40:24 11:35:26 INFO Pod/spark-operator-controller-767899d44f-clthp: Killing - Stopping container controller 2026/07/12 11:40:24 11:35:26 INFO Deployment/spark-operator-controller: ScalingReplicaSet - Scaled up replica set spark-operator-controller-767899d44f from 0 to 1 2026/07/12 11:40:24 11:35:26 INFO ReplicaSet/spark-operator-controller-767899d44f: SuccessfulCreate - Created pod: spark-operator-controller-767899d44f-mtx65 2026/07/12 11:40:24 === RESOURCE QUOTAS === 2026/07/12 11:40:24 No resource quotas found in monitored namespaces 2026/07/12 11:40:24 Classification for group 2: infrastructure/storage (code=1006, confidence=medium) 2026/07/12 11:40:24 Evidence: event Pod/workload-variant-autoscaler-controller-manager-5bd688895b-td5pc: FailedMount - MountVolume.SetUp failed for volume "epp-metrics-token" : secret "workload-variant-autoscaler-epp-metrics-token" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/workload-variant-autoscaler-controller-manager-5bd688895b-td5pc: FailedMount - MountVolume.SetUp failed for volume \"epp-metrics-token\" : secret \"workload-variant-autoscaler-epp-metrics-token\" not found"],"confidence":"medium"} 2026/07/12 11:40:24 Diagnostics complete for failed test: group 2 controller_test.go:753: Stopping: group 2 test failed. 2026/07/12 11:40:24 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/12 11:40:24 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/12 11:44:32 Circuit breaker: success recorded, resetting failure counter (was 1) 2026/07/12 11:53:13 TEST FAILURE DETECTED: group 4 2026/07/12 11:53:13 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 4) === 2026/07/12 11:53:14 === CLUSTER STATE === 2026/07/12 11:53:14 Node: ods-qe-psi-07-b4mqp-master-0 2026/07/12 11:53:14 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:53:14 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:53:14 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:53:14 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:53:14 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/12 11:53:14 Node: ods-qe-psi-07-b4mqp-master-1 2026/07/12 11:53:14 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:53:14 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:53:14 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:53:14 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:53:14 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/12 11:53:14 Node: ods-qe-psi-07-b4mqp-master-2 2026/07/12 11:53:14 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:53:14 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:53:14 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:53:14 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:53:14 Resources: 11500m CPU, 23450732Ki memory allocatable (of 12000m CPU, 24601708Ki memory capacity) 2026/07/12 11:53:14 Node: ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/12 11:53:14 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:53:14 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:53:14 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:53:14 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:53:14 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/12 11:53:14 Node: ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/12 11:53:14 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:53:14 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:53:14 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:53:14 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:53:14 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/12 11:53:14 Node: ods-qe-psi-07-b4mqp-worker-0-jd4pm 2026/07/12 11:53:14 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:53:14 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:53:14 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:53:14 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:53:14 Resources: 11500m CPU, 23452736Ki memory allocatable (of 12000m CPU, 24603712Ki memory capacity) 2026/07/12 11:53:14 Node: ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/12 11:53:14 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:53:14 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:53:14 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:53:14 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:53:14 Resources: 11500m CPU, 23452744Ki memory allocatable (of 12000m CPU, 24603720Ki memory capacity) 2026/07/12 11:53:14 Node: ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/12 11:53:14 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:53:14 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:53:14 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:53:14 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:53:14 Resources: 11500m CPU, 23450720Ki memory allocatable (of 12000m CPU, 24601696Ki memory capacity) 2026/07/12 11:53:14 === DEPLOYMENTS === 2026/07/12 11:53:14 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/12 11:53:14 === PODS === 2026/07/12 11:53:14 Failed to collect pod data: redhat-ods-applications/maas-controller-85bdcf9988-dtt7m: container manager not ready 2026/07/12 11:53:14 === OPERATOR STATUS === 2026/07/12 11:53:14 Operator deployment: 3/3 ready 2026/07/12 11:53:14 === DSCI STATUS === 2026/07/12 11:53:14 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/12 11:53:14 === DSC STATUS === 2026/07/12 11:53:14 DSC default-dsc: 2026/07/12 11:53:14 Ready: True - 2026/07/12 11:53:14 ProvisioningSucceeded: True - 2026/07/12 11:53:14 ComponentsReady: True - 2026/07/12 11:53:14 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/12 11:53:14 DashboardReady: False - Component ManagementState is set to Removed 2026/07/12 11:53:14 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/12 11:53:14 KserveLLMInferenceServiceDependencies: True - 2026/07/12 11:53:14 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/12 11:53:14 KserveReady: True - 2026/07/12 11:53:14 KueueReady: True - 2026/07/12 11:53:14 LLM-D-WVADependencies: True - 2026/07/12 11:53:14 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/12 11:53:14 ModelControllerReady: True - 2026/07/12 11:53:14 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/12 11:53:14 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/12 11:53:14 OGXReady: False - Component ManagementState is set to Removed 2026/07/12 11:53:14 RayReady: False - Component ManagementState is set to Removed 2026/07/12 11:53:14 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/12 11:53:14 TrainerReady: False - Component ManagementState is set to Removed 2026/07/12 11:53:14 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/12 11:53:14 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/12 11:53:14 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/12 11:53:14 === RECENT EVENTS === 2026/07/12 11:53:14 11:52:24 INFO Pod/maas-controller-85bdcf9988-dtt7m: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 621ms (621ms including waiting). Image size: 212866077 bytes. 2026/07/12 11:53:14 11:52:23 INFO Pod/maas-controller-85bdcf9988-dtt7m: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/12 11:53:14 11:49:45 WARN Pod/maas-controller-85bdcf9988-dtt7m: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-vz4sc); attempting to pull the image may not succeed. 2026/07/12 11:53:14 11:49:35 INFO Pod/maas-controller-85bdcf9988-dtt7m: Created - Created container: manager 2026/07/12 11:53:14 11:49:35 INFO Pod/maas-controller-85bdcf9988-dtt7m: Started - Started container manager 2026/07/12 11:53:14 11:49:35 INFO Pod/maas-controller-85bdcf9988-dtt7m: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 729ms (729ms including waiting). Image size: 212866077 bytes. 2026/07/12 11:53:14 === RESOURCE QUOTAS === 2026/07/12 11:53:14 No resource quotas found in monitored namespaces 2026/07/12 11:53:14 Classification for group 4: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/12 11:53:14 Evidence: container maas-controller-85bdcf9988-dtt7m/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-dtt7m_redhat-ods-applications(e017a851-ec77-425f-9b59-66954b4052b2) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-85bdcf9988-dtt7m/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-dtt7m_redhat-ods-applications(e017a851-ec77-425f-9b59-66954b4052b2)"],"confidence":"medium"} 2026/07/12 11:53:14 Diagnostics complete for failed test: group 4 controller_test.go:753: Stopping: group 4 test failed. 2026/07/12 11:53:14 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/12 11:53:14 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/12 11:54:25 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (1451.82s) === RUN TestOdhOperator 2026/07/12 11:31:03 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/12 11:55:26 TEST FAILURE DETECTED: components 2026/07/12 11:55:26 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/12 11:55:26 === CLUSTER STATE === 2026/07/12 11:55:26 Node: ods-qe-psi-07-b4mqp-master-0 2026/07/12 11:55:26 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:55:26 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:55:26 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:55:26 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:55:26 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/12 11:55:26 Node: ods-qe-psi-07-b4mqp-master-1 2026/07/12 11:55:26 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:55:26 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:55:26 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:55:26 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:55:26 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/12 11:55:26 Node: ods-qe-psi-07-b4mqp-master-2 2026/07/12 11:55:26 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:55:26 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:55:26 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:55:26 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:55:26 Resources: 11500m CPU, 23450732Ki memory allocatable (of 12000m CPU, 24601708Ki memory capacity) 2026/07/12 11:55:26 Node: ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/12 11:55:26 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:55:26 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:55:26 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:55:26 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:55:26 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/12 11:55:26 Node: ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/12 11:55:26 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:55:26 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:55:26 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:55:26 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:55:26 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/12 11:55:26 Node: ods-qe-psi-07-b4mqp-worker-0-jd4pm 2026/07/12 11:55:26 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:55:26 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:55:26 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:55:26 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:55:26 Resources: 11500m CPU, 23452736Ki memory allocatable (of 12000m CPU, 24603712Ki memory capacity) 2026/07/12 11:55:26 Node: ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/12 11:55:26 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:55:26 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:55:26 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:55:26 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:55:26 Resources: 11500m CPU, 23452744Ki memory allocatable (of 12000m CPU, 24603720Ki memory capacity) 2026/07/12 11:55:26 Node: ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/12 11:55:26 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 11:55:26 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 11:55:26 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 11:55:26 Condition Ready: True - kubelet is posting ready status 2026/07/12 11:55:26 Resources: 11500m CPU, 23450720Ki memory allocatable (of 12000m CPU, 24601696Ki memory capacity) 2026/07/12 11:55:26 === DEPLOYMENTS === 2026/07/12 11:55:26 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/12 11:55:26 === PODS === 2026/07/12 11:55:26 Failed to collect pod data: redhat-ods-applications/maas-controller-85bdcf9988-dtt7m: container manager not ready 2026/07/12 11:55:26 === OPERATOR STATUS === 2026/07/12 11:55:26 Operator deployment: 3/3 ready 2026/07/12 11:55:26 === DSCI STATUS === 2026/07/12 11:55:26 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/12 11:55:26 === DSC STATUS === 2026/07/12 11:55:26 DSC default-dsc: 2026/07/12 11:55:26 Ready: True - 2026/07/12 11:55:26 ProvisioningSucceeded: True - 2026/07/12 11:55:26 ComponentsReady: True - 2026/07/12 11:55:26 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/12 11:55:26 DashboardReady: False - Component ManagementState is set to Removed 2026/07/12 11:55:26 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/12 11:55:26 KserveLLMInferenceServiceDependencies: True - 2026/07/12 11:55:26 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/12 11:55:26 KserveReady: True - 2026/07/12 11:55:26 KueueReady: True - 2026/07/12 11:55:26 LLM-D-WVADependencies: True - 2026/07/12 11:55:26 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/12 11:55:26 ModelControllerReady: True - 2026/07/12 11:55:26 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/12 11:55:26 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/12 11:55:26 OGXReady: False - Component ManagementState is set to Removed 2026/07/12 11:55:26 RayReady: False - Component ManagementState is set to Removed 2026/07/12 11:55:26 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/12 11:55:26 TrainerReady: True - 2026/07/12 11:55:26 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/12 11:55:26 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/12 11:55:26 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/12 11:55:26 === RECENT EVENTS === 2026/07/12 11:55:26 11:55:10 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-wstth_39fff12c-aff7-4055-bb7e-b6bfcf8697af became leader 2026/07/12 11:55:26 11:55:09 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-8s6bp_d810e3d9-2bcc-4e6c-81b8-43aedaa00c4f became leader 2026/07/12 11:55:26 11:54:55 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-fw4f6_a8c7edad-d557-463b-9b03-c89dcd15141b became leader 2026/07/12 11:55:26 11:54:52 INFO Pod/kserve-controller-manager-6c67dcb69c-8s6bp: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 2.075s (2.075s including waiting). Image size: 238435613 bytes. 2026/07/12 11:55:26 11:54:52 INFO Pod/kserve-controller-manager-6c67dcb69c-8s6bp: Created - Created container: manager 2026/07/12 11:55:26 11:54:52 INFO Pod/kserve-controller-manager-6c67dcb69c-8s6bp: Started - Started container manager 2026/07/12 11:55:26 11:54:51 INFO Pod/llmisvc-controller-manager-5d65df97bb-wstth: Created - Created container: manager 2026/07/12 11:55:26 11:54:51 INFO Pod/llmisvc-controller-manager-5d65df97bb-wstth: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 661ms (661ms including waiting). Image size: 236366599 bytes. 2026/07/12 11:55:26 11:54:51 INFO Pod/llmisvc-controller-manager-5d65df97bb-wstth: Started - Started container manager 2026/07/12 11:55:26 11:54:50 INFO Pod/kserve-controller-manager-6c67dcb69c-8s6bp: AddedInterface - Add eth0 [10.131.3.58/23] from ovn-kubernetes 2026/07/12 11:55:26 11:54:50 INFO Pod/llmisvc-controller-manager-5d65df97bb-wstth: AddedInterface - Add eth0 [10.131.3.59/23] from ovn-kubernetes 2026/07/12 11:55:26 11:54:50 INFO Pod/llmisvc-controller-manager-5d65df97bb-wstth: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/12 11:55:26 11:54:50 WARN Pod/maas-controller-85bdcf9988-dtt7m: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-vz4sc); attempting to pull the image may not succeed. 2026/07/12 11:55:26 11:54:50 INFO Pod/kserve-controller-manager-6c67dcb69c-8s6bp: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/12 11:55:26 11:54:49 INFO Pod/llmisvc-controller-manager-5d65df97bb-wstth: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-wstth to ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/12 11:55:26 11:54:49 INFO Pod/kserve-controller-manager-6c67dcb69c-8s6bp: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-8s6bp to ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/12 11:55:26 11:54:49 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/12 11:55:26 11:54:49 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-8s6bp 2026/07/12 11:55:26 11:54:49 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-wstth 2026/07/12 11:55:26 11:54:49 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/12 11:55:26 11:54:39 INFO Pod/model-serving-api-6688cd56bd-plx24: Started - Started container server 2026/07/12 11:55:26 11:54:39 INFO Pod/model-serving-api-6688cd56bd-plx24: Created - Created container: server 2026/07/12 11:55:26 11:54:39 INFO Pod/model-serving-api-6688cd56bd-plx24: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/12 11:55:26 11:54:39 INFO Pod/model-serving-api-6688cd56bd-plx24: AddedInterface - Add eth0 [10.130.2.144/23] from ovn-kubernetes 2026/07/12 11:55:26 11:54:38 INFO Pod/model-serving-api-6688cd56bd-plx24: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-plx24 to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/12 11:55:26 11:54:38 INFO Pod/odh-model-controller-6849fc49d8-fw4f6: Started - Started container manager 2026/07/12 11:55:26 11:54:38 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/12 11:55:26 11:54:38 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-plx24 2026/07/12 11:55:26 11:54:38 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/12 11:55:26 11:54:38 INFO Pod/odh-model-controller-6849fc49d8-fw4f6: AddedInterface - Add eth0 [10.130.2.143/23] from ovn-kubernetes 2026/07/12 11:55:26 11:54:38 INFO Pod/odh-model-controller-6849fc49d8-fw4f6: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/12 11:55:26 11:54:38 INFO Pod/odh-model-controller-6849fc49d8-fw4f6: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 689ms (689ms including waiting). Image size: 238959091 bytes. 2026/07/12 11:55:26 11:54:38 INFO Pod/odh-model-controller-6849fc49d8-fw4f6: Created - Created container: manager 2026/07/12 11:55:26 11:54:38 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/12 11:55:26 11:54:37 WARN Pod/odh-model-controller-6849fc49d8-fw4f6: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/12 11:55:26 11:54:36 INFO Pod/odh-model-controller-6849fc49d8-fw4f6: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-fw4f6 to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/12 11:55:26 11:54:36 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-fw4f6 2026/07/12 11:55:26 11:54:36 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/12 11:55:26 11:54:26 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/12 11:55:26 11:54:26 INFO Pod/odh-model-controller-6849fc49d8-tlzgj: Killing - Stopping container manager 2026/07/12 11:55:26 11:54:26 WARN Pod/kserve-controller-manager-6c67dcb69c-zm44f: Unhealthy - Readiness probe failed: Get "http://10.130.2.142:8081/readyz": dial tcp 10.130.2.142:8081: connect: connection refused 2026/07/12 11:55:26 11:54:26 INFO Pod/kserve-controller-manager-6c67dcb69c-zm44f: Killing - Stopping container manager 2026/07/12 11:55:26 11:54:26 INFO Pod/llmisvc-controller-manager-5d65df97bb-h6jjm: Killing - Stopping container manager 2026/07/12 11:55:26 11:54:26 INFO Pod/model-serving-api-6688cd56bd-vdqj8: Killing - Stopping container server 2026/07/12 11:55:26 11:54:04 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-h6jjm_15a83ae0-ff4a-4c33-b73a-be8d0bb70f02 became leader 2026/07/12 11:55:26 11:54:01 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-zm44f_fc8c2546-e168-4d22-ac35-95b376cd0006 became leader 2026/07/12 11:55:26 11:53:44 INFO Pod/llmisvc-controller-manager-5d65df97bb-h6jjm: Started - Started container manager 2026/07/12 11:55:26 11:53:44 INFO Pod/kserve-controller-manager-6c67dcb69c-zm44f: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 707ms (707ms including waiting). Image size: 238435613 bytes. 2026/07/12 11:55:26 11:53:44 INFO Pod/llmisvc-controller-manager-5d65df97bb-h6jjm: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 823ms (823ms including waiting). Image size: 236366599 bytes. 2026/07/12 11:55:26 11:53:44 INFO Pod/llmisvc-controller-manager-5d65df97bb-h6jjm: Created - Created container: manager 2026/07/12 11:55:26 11:53:44 INFO Pod/kserve-controller-manager-6c67dcb69c-zm44f: Started - Started container manager 2026/07/12 11:55:26 11:53:44 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-tlzgj_d350d837-90e0-47cb-ab2b-5cc8c7ccc542 became leader 2026/07/12 11:55:26 11:53:44 INFO Pod/kserve-controller-manager-6c67dcb69c-zm44f: Created - Created container: manager 2026/07/12 11:55:26 11:53:43 INFO Pod/llmisvc-controller-manager-5d65df97bb-h6jjm: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-h6jjm to ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/12 11:55:26 11:53:43 INFO Pod/kserve-controller-manager-6c67dcb69c-zm44f: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-zm44f to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/12 11:55:26 11:53:43 INFO Pod/kserve-controller-manager-6c67dcb69c-zm44f: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/12 11:55:26 11:53:43 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/12 11:55:26 11:53:43 INFO Pod/kserve-controller-manager-6c67dcb69c-zm44f: AddedInterface - Add eth0 [10.130.2.142/23] from ovn-kubernetes 2026/07/12 11:55:26 11:53:43 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/12 11:55:26 11:53:43 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-h6jjm 2026/07/12 11:55:26 11:53:43 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-zm44f 2026/07/12 11:55:26 11:53:43 INFO Pod/llmisvc-controller-manager-5d65df97bb-h6jjm: AddedInterface - Add eth0 [10.131.3.57/23] from ovn-kubernetes 2026/07/12 11:55:26 11:53:43 INFO Pod/llmisvc-controller-manager-5d65df97bb-h6jjm: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/12 11:55:26 11:53:36 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-7ccfdd66-skq4h_8b994117-56b0-44e3-9467-86194e0e9129 became leader 2026/07/12 11:55:26 11:53:27 INFO Pod/model-serving-api-6688cd56bd-vdqj8: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-vdqj8 to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/12 11:55:26 11:53:27 INFO Pod/odh-model-controller-6849fc49d8-tlzgj: AddedInterface - Add eth0 [10.130.2.140/23] from ovn-kubernetes 2026/07/12 11:55:26 11:53:27 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-vdqj8 2026/07/12 11:55:26 11:53:27 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/12 11:55:26 11:53:27 INFO Pod/model-serving-api-6688cd56bd-vdqj8: Started - Started container server 2026/07/12 11:55:26 11:53:27 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/12 11:55:26 11:53:27 INFO Pod/model-serving-api-6688cd56bd-vdqj8: Created - Created container: server 2026/07/12 11:55:26 11:53:27 INFO Pod/model-serving-api-6688cd56bd-vdqj8: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/12 11:55:26 11:53:27 INFO Pod/model-serving-api-6688cd56bd-vdqj8: AddedInterface - Add eth0 [10.130.2.141/23] from ovn-kubernetes 2026/07/12 11:55:26 11:53:27 INFO Pod/odh-model-controller-6849fc49d8-tlzgj: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/12 11:55:26 11:53:27 INFO Pod/odh-model-controller-6849fc49d8-tlzgj: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 663ms (663ms including waiting). Image size: 238959091 bytes. 2026/07/12 11:55:26 11:53:27 INFO Pod/odh-model-controller-6849fc49d8-tlzgj: Created - Created container: manager 2026/07/12 11:55:26 11:53:27 INFO Pod/odh-model-controller-6849fc49d8-tlzgj: Started - Started container manager 2026/07/12 11:55:26 11:53:26 WARN Pod/odh-model-controller-6849fc49d8-tlzgj: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/12 11:55:26 11:53:25 INFO Pod/odh-model-controller-6849fc49d8-tlzgj: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-tlzgj to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/12 11:55:26 11:53:25 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/12 11:55:26 11:53:25 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-tlzgj 2026/07/12 11:55:26 11:53:21 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-skq4h: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-trainer-rhel9@sha256:59ea523e2df9c1e705c755308ac11420682081cc63d54890e2b15fbd7d8ba570" in 1.796s (1.796s including waiting). Image size: 157432248 bytes. 2026/07/12 11:55:26 11:53:21 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-skq4h: Started - Started container manager 2026/07/12 11:55:26 11:53:21 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-skq4h: Created - Created container: manager 2026/07/12 11:55:26 11:53:20 WARN Pod/kserve-controller-manager-6c67dcb69c-sd5mx: Unhealthy - Readiness probe failed: Get "http://10.130.2.137:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026/07/12 11:55:26 11:53:19 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-skq4h: AddedInterface - Add eth0 [10.130.2.139/23] from ovn-kubernetes 2026/07/12 11:55:26 11:53:19 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-skq4h: Pulling - Pulling image "registry.redhat.io/rhoai/odh-trainer-rhel9@sha256:59ea523e2df9c1e705c755308ac11420682081cc63d54890e2b15fbd7d8ba570" 2026/07/12 11:55:26 11:53:18 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-skq4h: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-7ccfdd66-skq4h to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/12 11:55:26 11:53:18 INFO ReplicaSet/kubeflow-trainer-controller-manager-7ccfdd66: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-7ccfdd66-skq4h 2026/07/12 11:55:26 11:53:18 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-7ccfdd66 from 0 to 1 2026/07/12 11:55:26 11:53:15 INFO Pod/odh-model-controller-6849fc49d8-hfnr8: Killing - Stopping container manager 2026/07/12 11:55:26 11:53:15 INFO Pod/model-serving-api-6688cd56bd-sxh75: Killing - Stopping container server 2026/07/12 11:55:26 11:53:15 INFO Pod/kserve-controller-manager-6c67dcb69c-sd5mx: Killing - Stopping container manager 2026/07/12 11:55:26 11:53:15 INFO Pod/llmisvc-controller-manager-5d65df97bb-q98sq: Killing - Stopping container manager 2026/07/12 11:55:26 11:52:24 INFO Pod/maas-controller-85bdcf9988-dtt7m: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 621ms (621ms including waiting). Image size: 212866077 bytes. 2026/07/12 11:55:26 11:52:23 INFO Pod/maas-controller-85bdcf9988-dtt7m: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/12 11:55:26 === RESOURCE QUOTAS === 2026/07/12 11:55:26 No resource quotas found in monitored namespaces 2026/07/12 11:55:26 Classification for components: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/12 11:55:26 Evidence: container maas-controller-85bdcf9988-dtt7m/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-dtt7m_redhat-ods-applications(e017a851-ec77-425f-9b59-66954b4052b2) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-85bdcf9988-dtt7m/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-dtt7m_redhat-ods-applications(e017a851-ec77-425f-9b59-66954b4052b2)"],"confidence":"medium"} 2026/07/12 11:55:26 Diagnostics complete for failed test: components controller_test.go:484: Stopping: components test failed. 2026/07/12 11:55:26 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/12 11:55:26 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/12 11:55:27 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (1464.04s)