=== RUN TestOdhOperator/components/group_2/kueue/Validate_component_removed_to_unmanaged_transition helper_test.go:150: Setting Kueue component (default-kueue) to Removed mode to start with a clean state. helper_test.go:150: Cleaning up any existing Kueue test resources. helper_test.go:150: Cleaning up Kueue cluster-scoped resources helper_test.go:150: Will attempt to delete 2 Kueue resources helper_test.go:150: Attempting to delete ClusterQueue /default helper_test.go:150: Successfully processed deletion of ClusterQueue /default helper_test.go:150: Attempting to delete Kueue /cluster helper_test.go:150: Successfully processed deletion of Kueue /cluster helper_test.go:150: Creating test namespace (test-kueue-managed-d98ah2ns6vus7b145rqg) with Kueue management annotation. helper_test.go:150: Creating Kueue ConfigMap (kueue-manager-config) to verify default Kueue resource is created correctly. helper_test.go:150: Creating Kueue ConfigMap (kueue-manager-config) in namespace (redhat-ods-applications). helper_test.go:150: Updating the management state of the component in the DataScienceCluster to Unmanaged (expecting NotReady initially due to missing operator). helper_test.go:150: Ensuring LWS Operator is installed (namespace=openshift-lws-operator, name=leader-worker-set). helper_test.go:150: Ensuring LeaderWorkerSetOperator CR exists (namespace=openshift-lws-operator, name=cluster). helper_test.go:150: Ensuring OCP Kueue Operator is installed (namespace=openshift-kueue-operator, name=kueue-operator). helper_test.go:150: Verifying the component in the DataScienceCluster transitions to Ready state. helper_test.go:150: Verifying Kueue configuration (cluster) is created with all expected frameworks. testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_2/kueue/Validate_component_removed_to_unmanaged_transition (31.65s) === RUN TestOdhOperator/components/group_2/kueue === PAUSE TestOdhOperator/components/group_2/kueue === CONT TestOdhOperator/components/group_2/kueue test_context_test.go:223: Timed out after 10.001s. The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:211 with: Expected resource 'cluster' of kind 'Kueue' to exist, but it was not found or could not be retrieved. Expected <*unstructured.Unstructured | 0x0>: nil not to be nil test_context_test.go:1106: cleanup: best-effort delete failed for /, Kind= redhat-ods-applications/kueue-manager-config: Object 'Kind' is missing in 'unstructured object has no kind' --- FAIL: TestOdhOperator/components/group_2/kueue (32.94s) === RUN TestOdhOperator/components/group_2 --- FAIL: TestOdhOperator/components/group_2 (0.00s) === RUN TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable helper_test.go:150: Verifying Tenant models-as-a-service/default-tenant is present before disable helper_test.go:150: Disabling MaaS subcomponent (setting to Removed) helper_test.go:150: Waiting for Tenant models-as-a-service/default-tenant to be deleted testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable (430.29s) === RUN TestOdhOperator/components/group_4/modelsasservice === PAUSE TestOdhOperator/components/group_4/modelsasservice === CONT TestOdhOperator/components/group_4/modelsasservice 2026-07-10T08:16:48Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.19.18"}} controller_test.go:753: Creating MaaS PostgreSQL instance in namespace redhat-ods-applications controller_test.go:753: MaaS PostgreSQL instance and maas-db-config secret created in namespace redhat-ods-applications controller_test.go:753: Creating MaaS GatewayClass and Gateway: openshift-ingress/maas-default-gateway controller_test.go:753: Using hostname=maas.apps.ods-qe-psi-07.osp.rh-ods.com, certSecret=router-certs-default controller_test.go:753: MaaS Gateway openshift-ingress/maas-default-gateway created with HTTP+HTTPS listeners test_context_test.go:615: Timed out after 420.009s. The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1824 with: Expected resource list 'models-as-a-service/default-tenant' of kind 'Tenant' to be empty, but it contains resources. Expected <[]unstructured.Unstructured | len:1, cap:1>: [ { Object: { "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "kind": <string>"Tenant", "metadata": <map[string]interface {} | len:8>{ "namespace": <string>"models-as-a-service", "ownerReferences": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:4>{ "kind": <string>"Config", "name": <string>"default", "uid": <string>"9a2f5b87-4aee-4430-919a-9c319b6db2e2", "apiVersion": <string>"maas.opendatahub.io/v1alpha1", }, ], "resourceVersion": <string>"387826405", "uid": <string>"a94058e1-bc40-409f-8df8-78a53d1e5839", "creationTimestamp": <string>"2026-07-10T08:17:49Z", "generation": <int64>1, "managedFields": <[]interface {} | len:2, cap:2>[ <map[string]interface {} | len:6>{ "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:2>{ "f:metadata": <map[string]interface {} | len:1>{ "f:ownerReferences": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "k:{\"uid\":\"9a2f5b87-4aee-4430-919a-9c319b6db2e2\"}": <map[string]interface {} | len:0>{}, }, }, "f:spec": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "f:gatewayRef": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:name": <map[string]interface {} | len:0>{}, "f:namespace": <map[string]interface {} | len:0>{}, }, }, }, "manager": <string>"manager", "operation": <string>"Update", "time": <string>"2026-07-10T08:17:49Z", "apiVersion": <string>"maas.opendatahub.io/v1alpha1", }, <map[string]interface {} | len:7>{ "manager": <string>"manager", "operation": <string>"Update", "subresource": <string>"status", "time": <string>"2026-07-10T08:18:12Z", "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:1>{ "f:status": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:conditions": <map[string]interface {} | len:0>{}, "f:phase": <map[string]interface {} | len:0>{}, }, }, }, ], "name": <string>"default-tenant", }, "spec": <map[string]interface {} | len:1>{ "gatewayRef": <map[string]interface {} | len:2>{ "namespace": <string>"openshift-ingress", "name": <string>"maas-default-gateway", }, }, "status": <map[string]interface {} | len:2>{ "conditions": <[]interface {} | len:5, cap:8>[ <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-07-10T08:17:51Z", "message": <string>"AuthConfig CRD (Kuadrant) is available on the cluster", "observedGeneration": <int64>1, "reason": <string>"DependenciesMet", "status": <string>"True", "type": <string>"DependenciesAvailable", }, <map[string]interface {} | len:6>{ "type": <string>"MaaSPrerequisitesAvailable", "lastTransitionTime": <string>"2026-07-10T08:17:51Z", "message": <string>"All prerequisites are satisfied", "observedGeneration": <int64>1, "reason": <string>"PrerequisitesMet", "status": <string>"True", }, <map[string]interface {} | len:6>{ "observedGeneration": <int64>1, "reason": <string>"PrerequisitesMet", "status": <string>"False", "type": <string>"Degraded", "lastTransitionTime": <string>"2026-07-10T08:17:51Z", "message": <string>"All prerequisites are satisfied", }, <map[string]interface {} | len:6>{ "reason": <string>"DeploymentsReady", "status": <string>"True", "type": <string>"DeploymentsAvailable", "lastTransitionTime": <string>"2026-07-10T08:18:12Z", "message": <string>"maas-api deployment is available", "observedGeneration": <int64>1, }, <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-07-10T08:18:12Z", "message": <string>"MaaS platform manifests applied and maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"Reconciled", "status": <string>"True", "type": <string>"Ready", }, ], "phase": <string>"Active", }, }, }, ] to be empty --- FAIL: TestOdhOperator/components/group_4/modelsasservice (522.51s) === RUN TestOdhOperator/components/group_4 --- FAIL: TestOdhOperator/components/group_4 (0.00s) === RUN TestOdhOperator/components 2026/07/10 08:12:35 TEST FAILURE DETECTED: group 2 2026/07/10 08:12:35 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 2) === 2026/07/10 08:12:37 === CLUSTER STATE === 2026/07/10 08:12:37 Node: ods-qe-psi-07-b4mqp-master-0 2026/07/10 08:12:37 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 08:12:37 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 08:12:37 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 08:12:37 Condition Ready: True - kubelet is posting ready status 2026/07/10 08:12:37 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/10 08:12:37 Node: ods-qe-psi-07-b4mqp-master-1 2026/07/10 08:12:37 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 08:12:37 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 08:12:37 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 08:12:37 Condition Ready: True - kubelet is posting ready status 2026/07/10 08:12:37 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/10 08:12:37 Node: ods-qe-psi-07-b4mqp-master-2 2026/07/10 08:12:37 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 08:12:37 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 08:12:37 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 08:12:37 Condition Ready: True - kubelet is posting ready status 2026/07/10 08:12:37 Resources: 11500m CPU, 23450732Ki memory allocatable (of 12000m CPU, 24601708Ki memory capacity) 2026/07/10 08:12:37 Node: ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/10 08:12:37 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 08:12:37 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 08:12:37 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 08:12:37 Condition Ready: True - kubelet is posting ready status 2026/07/10 08:12:37 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/10 08:12:37 Node: ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/10 08:12:37 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 08:12:37 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 08:12:37 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 08:12:37 Condition Ready: True - kubelet is posting ready status 2026/07/10 08:12:37 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/10 08:12:37 Node: ods-qe-psi-07-b4mqp-worker-0-jd4pm 2026/07/10 08:12:37 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 08:12:37 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 08:12:37 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 08:12:37 Condition Ready: True - kubelet is posting ready status 2026/07/10 08:12:37 Resources: 11500m CPU, 23452736Ki memory allocatable (of 12000m CPU, 24603712Ki memory capacity) 2026/07/10 08:12:37 Node: ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/10 08:12:37 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 08:12:37 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 08:12:37 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 08:12:37 Condition Ready: True - kubelet is posting ready status 2026/07/10 08:12:37 Resources: 11500m CPU, 23452744Ki memory allocatable (of 12000m CPU, 24603720Ki memory capacity) 2026/07/10 08:12:37 Node: ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/10 08:12:37 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 08:12:37 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 08:12:37 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 08:12:37 Condition Ready: True - kubelet is posting ready status 2026/07/10 08:12:37 Resources: 11500m CPU, 23450720Ki memory allocatable (of 12000m CPU, 24601696Ki memory capacity) 2026/07/10 08:12:37 === DEPLOYMENTS === 2026/07/10 08:12:37 Namespace: redhat-ods-applications 2026/07/10 08:12:37 Deployment maas-api: 1/1 (READY) 2026/07/10 08:12:37 Deployment maas-controller: 1/1 (READY) 2026/07/10 08:12:37 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/10 08:12:37 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/10 08:12:37 Namespace: kube-system 2026/07/10 08:12:37 === PODS === 2026/07/10 08:12:37 Namespace redhat-ods-applications: no problematic pods found 2026/07/10 08:12:37 Namespace kube-system: no problematic pods found 2026/07/10 08:12:37 === OPERATOR STATUS === 2026/07/10 08:12:37 Operator deployment: 3/3 ready 2026/07/10 08:12:37 === DSCI STATUS === 2026/07/10 08:12:37 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/10 08:12:37 === DSC STATUS === 2026/07/10 08:12:37 Failed to collect DSC data: default-dsc unhealthy conditions: ModelsAsServiceReady=False 2026/07/10 08:12:37 === RECENT EVENTS === 2026/07/10 08:12:37 08:12:27 INFO Pod/model-registry-operator-controller-manager-7f75765c5b-tct75: Killing - Stopping container manager 2026/07/10 08:12:37 08:12:27 INFO Pod/workload-variant-autoscaler-controller-manager-7967c6f77c-vgz7n: Killing - Stopping container manager 2026/07/10 08:12:37 08:12:27 INFO Pod/odh-model-controller-867855946f-htxgg: Killing - Stopping container manager 2026/07/10 08:12:37 08:12:27 INFO Pod/model-serving-api-6688cd56bd-m96wb: Killing - Stopping container server 2026/07/10 08:12:37 08:12:26 INFO Pod/kserve-controller-manager-7cd6f695d9-qdsm2: Killing - Stopping container manager 2026/07/10 08:12:37 08:12:26 INFO Pod/llmisvc-controller-manager-57b755898c-lsn7k: Killing - Stopping container manager 2026/07/10 08:12:37 08:12:25 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - ServiceMonitor controller-manager-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 08:12:37 08:12:25 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 08:12:37 08:12:12 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-867855946f-htxgg_08a867b7-6bb6-4c44-b197-acb7d09b594e became leader 2026/07/10 08:12:37 08:12:11 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-7967c6f77c-vgz7n_eef522fb-f3af-45d4-b2c8-56d05a42bb07 became leader 2026/07/10 08:12:37 08:12:11 INFO Pod/workload-variant-autoscaler-controller-manager-7967c6f77c-vgz7n: Started - Started container manager 2026/07/10 08:12:37 08:12:11 INFO Pod/workload-variant-autoscaler-controller-manager-7967c6f77c-vgz7n: Created - Created container: manager 2026/07/10 08:12:37 08:12:10 INFO Pod/workload-variant-autoscaler-controller-manager-7967c6f77c-vgz7n: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-7967c6f77c-vgz7n to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/10 08:12:37 08:12:10 INFO Pod/workload-variant-autoscaler-controller-manager-7967c6f77c-vgz7n: Pulled - Container image "registry.redhat.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:4bbc5cf2ccade00f803ab5350ccae186d638075ed56244c344808b51554b1d0b" already present on machine 2026/07/10 08:12:37 08:12:10 INFO Pod/workload-variant-autoscaler-controller-manager-7967c6f77c-vgz7n: AddedInterface - Add eth0 [10.131.0.178/23] from ovn-kubernetes 2026/07/10 08:12:37 08:12:10 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-7967c6f77c from 0 to 1 2026/07/10 08:12:37 08:12:10 INFO Pod/workload-variant-autoscaler-controller-manager-7967c6f77c-qlgpp: Killing - Stopping container manager 2026/07/10 08:12:37 08:12:10 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-7967c6f77c: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-7967c6f77c-vgz7n 2026/07/10 08:12:37 08:12:10 INFO Pod/workload-variant-autoscaler-controller-manager-7967c6f77c-9bj59: Killing - Stopping container manager 2026/07/10 08:12:37 08:11:56 INFO Pod/odh-model-controller-867855946f-htxgg: Started - Started container manager 2026/07/10 08:12:37 08:11:56 INFO Pod/odh-model-controller-867855946f-htxgg: Created - Created container: manager 2026/07/10 08:12:37 08:11:56 INFO Pod/odh-model-controller-867855946f-htxgg: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 774ms (774ms including waiting). Image size: 238959091 bytes. 2026/07/10 08:12:37 08:11:55 INFO Pod/odh-model-controller-867855946f-htxgg: AddedInterface - Add eth0 [10.131.0.177/23] from ovn-kubernetes 2026/07/10 08:12:37 08:11:55 INFO Pod/odh-model-controller-867855946f-htxgg: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/10 08:12:37 08:11:54 INFO Pod/odh-model-controller-867855946f-htxgg: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-867855946f-htxgg to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/10 08:12:37 08:11:54 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-867855946f from 0 to 1 2026/07/10 08:12:37 08:11:54 INFO ReplicaSet/odh-model-controller-867855946f: SuccessfulCreate - Created pod: odh-model-controller-867855946f-htxgg 2026/07/10 08:12:37 08:11:54 INFO Pod/odh-model-controller-867855946f-qdh4w: Killing - Stopping container manager 2026/07/10 08:12:37 08:11:54 INFO Pod/odh-model-controller-867855946f-jv4wn: Killing - Stopping container manager 2026/07/10 08:12:37 08:11:42 INFO Pod/model-serving-api-6688cd56bd-m96wb: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" in 1.598s (1.598s including waiting). Image size: 172848429 bytes. 2026/07/10 08:12:37 08:11:42 INFO Pod/model-serving-api-6688cd56bd-m96wb: Created - Created container: server 2026/07/10 08:12:37 08:11:42 INFO Pod/model-serving-api-6688cd56bd-m96wb: Started - Started container server 2026/07/10 08:12:37 08:11:40 INFO Pod/model-serving-api-6688cd56bd-m96wb: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" 2026/07/10 08:12:37 08:11:40 INFO Pod/model-serving-api-6688cd56bd-m96wb: AddedInterface - Add eth0 [10.131.0.176/23] from ovn-kubernetes 2026/07/10 08:12:37 08:11:39 INFO Pod/model-serving-api-6688cd56bd-m96wb: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-m96wb to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/10 08:12:37 08:11:39 INFO Pod/model-serving-api-6688cd56bd-qg44q: Killing - Stopping container server 2026/07/10 08:12:37 08:11:39 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-m96wb 2026/07/10 08:12:37 08:11:39 INFO Pod/model-serving-api-6688cd56bd-pqm8g: Killing - Stopping container server 2026/07/10 08:12:37 08:11:39 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/10 08:12:37 08:10:41 INFO Pod/workload-variant-autoscaler-controller-manager-7967c6f77c-9bj59: Started - Started container manager 2026/07/10 08:12:37 08:10:41 INFO Pod/workload-variant-autoscaler-controller-manager-7967c6f77c-9bj59: Created - Created container: manager 2026/07/10 08:12:37 08:10:40 INFO Pod/workload-variant-autoscaler-controller-manager-7967c6f77c-9bj59: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:4bbc5cf2ccade00f803ab5350ccae186d638075ed56244c344808b51554b1d0b" in 1.741s (1.741s including waiting). Image size: 195240093 bytes. 2026/07/10 08:12:37 08:10:39 INFO Pod/workload-variant-autoscaler-controller-manager-7967c6f77c-9bj59: AddedInterface - Add eth0 [10.130.2.54/23] from ovn-kubernetes 2026/07/10 08:12:37 08:10:39 INFO Pod/workload-variant-autoscaler-controller-manager-7967c6f77c-9bj59: Pulling - Pulling image "registry.redhat.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:4bbc5cf2ccade00f803ab5350ccae186d638075ed56244c344808b51554b1d0b" 2026/07/10 08:12:37 08:10:38 INFO Pod/workload-variant-autoscaler-controller-manager-7967c6f77c-9bj59: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-7967c6f77c-9bj59 to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/10 08:12:37 08:10:38 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-7967c6f77c: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-7967c6f77c-9bj59 2026/07/10 08:12:37 08:10:38 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-7967c6f77c from 1 to 2 2026/07/10 08:12:37 08:10:19 INFO Pod/odh-model-controller-867855946f-qdh4w: Started - Started container manager 2026/07/10 08:12:37 08:10:19 INFO Pod/odh-model-controller-867855946f-qdh4w: Created - Created container: manager 2026/07/10 08:12:37 08:10:19 INFO Pod/odh-model-controller-867855946f-qdh4w: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 708ms (708ms including waiting). Image size: 238959091 bytes. 2026/07/10 08:12:37 08:10:18 INFO Pod/odh-model-controller-867855946f-qdh4w: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-867855946f-qdh4w to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/10 08:12:37 08:10:18 INFO Pod/odh-model-controller-867855946f-qdh4w: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/10 08:12:37 08:10:18 INFO Pod/odh-model-controller-867855946f-qdh4w: AddedInterface - Add eth0 [10.131.0.175/23] from ovn-kubernetes 2026/07/10 08:12:37 08:10:18 INFO ReplicaSet/odh-model-controller-867855946f: SuccessfulCreate - Created pod: odh-model-controller-867855946f-qdh4w 2026/07/10 08:12:37 08:10:18 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-867855946f from 1 to 2 2026/07/10 08:12:37 08:10:00 INFO Pod/model-serving-api-6688cd56bd-pqm8g: Created - Created container: server 2026/07/10 08:12:37 08:10:00 INFO Pod/model-serving-api-6688cd56bd-pqm8g: Started - Started container server 2026/07/10 08:12:37 08:10:00 INFO Pod/model-serving-api-6688cd56bd-pqm8g: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" in 1.707s (1.707s including waiting). Image size: 172848429 bytes. 2026/07/10 08:12:37 08:09:58 INFO Pod/model-serving-api-6688cd56bd-pqm8g: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-pqm8g to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/10 08:12:37 08:09:58 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-pqm8g 2026/07/10 08:12:37 08:09:58 INFO Pod/model-serving-api-6688cd56bd-pqm8g: AddedInterface - Add eth0 [10.130.2.53/23] from ovn-kubernetes 2026/07/10 08:12:37 08:09:58 INFO Pod/model-serving-api-6688cd56bd-pqm8g: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" 2026/07/10 08:12:37 08:09:58 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 1 to 2 2026/07/10 08:12:37 08:09:36 INFO Pod/maas-api-645b759879-mzknt: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-api-rhel9@sha256:a143ad05859340f15a0426ca87ae1f26541c961b73e1991f9e8738d2589e60da" in 748ms (748ms including waiting). Image size: 175014302 bytes. 2026/07/10 08:12:37 08:09:36 INFO Pod/maas-api-645b759879-mzknt: Created - Created container: maas-api 2026/07/10 08:12:37 08:09:36 INFO Pod/maas-api-645b759879-mzknt: Started - Started container maas-api 2026/07/10 08:12:37 08:09:35 INFO Pod/maas-api-645b759879-mzknt: Scheduled - Successfully assigned redhat-ods-applications/maas-api-645b759879-mzknt to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/10 08:12:37 08:09:35 INFO Pod/maas-api-645b759879-mzknt: AddedInterface - Add eth0 [10.130.2.50/23] from ovn-kubernetes 2026/07/10 08:12:37 08:09:35 INFO Pod/maas-api-645b759879-mzknt: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-api-rhel9@sha256:a143ad05859340f15a0426ca87ae1f26541c961b73e1991f9e8738d2589e60da" 2026/07/10 08:12:37 08:09:35 INFO Deployment/maas-api: ScalingReplicaSet - Scaled up replica set maas-api-645b759879 from 0 to 1 2026/07/10 08:12:37 08:09:35 INFO ReplicaSet/maas-api-645b759879: SuccessfulCreate - Created pod: maas-api-645b759879-mzknt 2026/07/10 08:12:37 08:09:35 WARN ReplicaSet/maas-api-645b759879: FailedCreate - Error creating: pods "maas-api-645b759879-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 2026/07/10 08:12:37 08:09:20 INFO Pod/workload-variant-autoscaler-controller-manager-7967c6f77c-qlgpp: Started - Started container manager 2026/07/10 08:12:37 08:09:20 INFO Pod/workload-variant-autoscaler-controller-manager-7967c6f77c-qlgpp: Created - Created container: manager 2026/07/10 08:12:37 08:09:20 INFO Pod/workload-variant-autoscaler-controller-manager-7967c6f77c-qlgpp: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:4bbc5cf2ccade00f803ab5350ccae186d638075ed56244c344808b51554b1d0b" in 11.242s (11.242s including waiting). Image size: 195240093 bytes. 2026/07/10 08:12:37 08:09:20 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-7967c6f77c-qlgpp_db693bba-fdc0-4d60-97bd-ed68edb91f65 became leader 2026/07/10 08:12:37 08:09:19 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-57b755898c-lsn7k_395bd015-9366-47b7-aa53-7775d7da77f5 became leader 2026/07/10 08:12:37 08:09:18 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-7cd6f695d9-qdsm2_f89fa7b4-06d6-4a99-85ae-5190fa861a67 became leader 2026/07/10 08:12:37 08:09:09 INFO Pod/workload-variant-autoscaler-controller-manager-7967c6f77c-qlgpp: Pulling - Pulling image "registry.redhat.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:4bbc5cf2ccade00f803ab5350ccae186d638075ed56244c344808b51554b1d0b" 2026/07/10 08:12:37 08:09:09 INFO Pod/workload-variant-autoscaler-controller-manager-7967c6f77c-qlgpp: AddedInterface - Add eth0 [10.131.0.173/23] from ovn-kubernetes 2026/07/10 08:12:37 08:09:08 WARN ServiceMonitor/workload-variant-autoscaler-controller-manager-metrics-monitor: InvalidConfiguration - ServiceMonitor workload-variant-autoscaler-controller-manager-metrics-monitor was rejected due to invalid configuration: unable to get secret "workload-variant-autoscaler-controller-manager-token": secrets "workload-variant-autoscaler-controller-manager-token" not found 2026/07/10 08:12:37 08:09:07 INFO Pod/workload-variant-autoscaler-controller-manager-7967c6f77c-qlgpp: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-7967c6f77c-qlgpp to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/10 08:12:37 08:09:07 WARN Pod/workload-variant-autoscaler-controller-manager-7967c6f77c-qlgpp: FailedMount - MountVolume.SetUp failed for volume "epp-metrics-token" : secret "workload-variant-autoscaler-epp-metrics-token" not found 2026/07/10 08:12:37 08:09:06 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-7967c6f77c from 0 to 1 2026/07/10 08:12:37 08:09:06 WARN ReplicaSet/workload-variant-autoscaler-controller-manager-7967c6f77c: FailedCreate - Error creating: pods "workload-variant-autoscaler-controller-manager-7967c6f77c-" is forbidden: error looking up service account redhat-ods-applications/workload-variant-autoscaler-controller-manager: serviceaccount "workload-variant-autoscaler-controller-manager" not found 2026/07/10 08:12:37 08:09:06 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-7967c6f77c: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-7967c6f77c-qlgpp 2026/07/10 08:12:37 08:09:03 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-867855946f-jv4wn_294201ce-24bb-443f-94f9-72b5659baf9c became leader 2026/07/10 08:12:37 08:09:03 INFO Lease/85f368d1.opendatahub.io: LeaderElection - model-registry-operator-controller-manager-7f75765c5b-tct75_3cb1f6e5-ccc9-46d2-be56-9884681a6596 became leader 2026/07/10 08:12:37 08:09:03 INFO Lease/maas-controller.models-as-a-service.opendatahub.io: LeaderElection - maas-controller-74bc958949-kf25x_d62b4317-18a3-4b6b-b8dd-f3d4e563e5f3 became leader 2026/07/10 08:12:37 08:09:02 INFO Pod/kserve-controller-manager-7cd6f695d9-qdsm2: Started - Started container manager 2026/07/10 08:12:37 08:09:02 INFO Pod/kserve-controller-manager-7cd6f695d9-qdsm2: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:faa11de719b196594cea0673067e7f28fa3e330ad5181c0abe0e899e585fb009" in 2.778s (2.778s including waiting). Image size: 238435545 bytes. 2026/07/10 08:12:37 08:09:02 INFO Pod/llmisvc-controller-manager-57b755898c-lsn7k: Created - Created container: manager 2026/07/10 08:12:37 08:09:02 INFO Pod/llmisvc-controller-manager-57b755898c-lsn7k: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:378244503cc7d4569e676b0bc4b4abb2ce6e3c5f5f879d832464e0e7dc558906" in 2.702s (2.702s including waiting). Image size: 236366528 bytes. 2026/07/10 08:12:37 08:09:02 INFO Pod/llmisvc-controller-manager-57b755898c-lsn7k: Started - Started container manager 2026/07/10 08:12:37 08:09:02 INFO Pod/kserve-controller-manager-7cd6f695d9-qdsm2: Created - Created container: manager 2026/07/10 08:12:37 08:08:59 INFO Pod/llmisvc-controller-manager-57b755898c-lsn7k: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-57b755898c-lsn7k to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/10 08:12:37 08:08:59 INFO Pod/kserve-controller-manager-7cd6f695d9-qdsm2: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-7cd6f695d9-qdsm2 to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/10 08:12:37 08:08:59 INFO Pod/kserve-controller-manager-7cd6f695d9-qdsm2: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:faa11de719b196594cea0673067e7f28fa3e330ad5181c0abe0e899e585fb009" 2026/07/10 08:12:37 08:08:59 INFO Pod/kserve-controller-manager-7cd6f695d9-qdsm2: AddedInterface - Add eth0 [10.129.3.202/23] from ovn-kubernetes 2026/07/10 08:12:37 08:08:59 INFO Pod/llmisvc-controller-manager-57b755898c-lsn7k: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:378244503cc7d4569e676b0bc4b4abb2ce6e3c5f5f879d832464e0e7dc558906" 2026/07/10 08:12:37 08:08:59 INFO ReplicaSet/kserve-controller-manager-7cd6f695d9: SuccessfulCreate - Created pod: kserve-controller-manager-7cd6f695d9-qdsm2 2026/07/10 08:12:37 08:08:59 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7cd6f695d9 from 0 to 1 2026/07/10 08:12:37 08:08:59 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-57b755898c from 0 to 1 2026/07/10 08:12:37 08:08:59 INFO ReplicaSet/llmisvc-controller-manager-57b755898c: SuccessfulCreate - Created pod: llmisvc-controller-manager-57b755898c-lsn7k 2026/07/10 08:12:37 08:08:59 INFO Pod/llmisvc-controller-manager-57b755898c-lsn7k: AddedInterface - Add eth0 [10.129.3.203/23] from ovn-kubernetes 2026/07/10 08:12:37 08:08:47 INFO Pod/odh-model-controller-867855946f-jv4wn: Created - Created container: manager 2026/07/10 08:12:37 08:08:47 INFO Pod/model-registry-operator-controller-manager-7f75765c5b-tct75: Created - Created container: manager 2026/07/10 08:12:37 08:08:47 INFO Pod/odh-model-controller-867855946f-jv4wn: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 2.649s (2.649s including waiting). Image size: 238959091 bytes. 2026/07/10 08:12:37 08:08:47 INFO Pod/model-serving-api-6688cd56bd-qg44q: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/10 08:12:37 08:08:47 INFO Pod/model-serving-api-6688cd56bd-qg44q: Created - Created container: server 2026/07/10 08:12:37 08:08:47 INFO Pod/model-registry-operator-controller-manager-7f75765c5b-tct75: Started - Started container manager 2026/07/10 08:12:37 08:08:47 INFO Pod/odh-model-controller-867855946f-jv4wn: Started - Started container manager 2026/07/10 08:12:37 08:08:47 INFO Pod/maas-controller-74bc958949-kf25x: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:521fc9d9f6139ed08f80315acbdc95b68b90b12c3e866c76e9141b3e23e2fc0b" in 3.223s (3.223s including waiting). Image size: 212878878 bytes. 2026/07/10 08:12:37 08:08:47 INFO Pod/maas-controller-74bc958949-kf25x: Created - Created container: manager 2026/07/10 08:12:37 08:08:47 INFO Pod/maas-controller-74bc958949-kf25x: Started - Started container manager 2026/07/10 08:12:37 08:08:47 INFO Pod/model-serving-api-6688cd56bd-qg44q: Started - Started container server 2026/07/10 08:12:37 08:08:47 INFO Pod/model-registry-operator-controller-manager-7f75765c5b-tct75: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-registry-operator-rhel9@sha256:ebcec5c43468283ed3cbab50530f939166f8f3e8c9d306e91a8107d05d7fa9d4" in 2.546s (2.546s including waiting). Image size: 211356146 bytes. 2026/07/10 08:12:37 08:08:46 INFO Pod/model-serving-api-6688cd56bd-qg44q: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-qg44q to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/10 08:12:37 08:08:46 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-qg44q 2026/07/10 08:12:37 08:08:46 INFO Pod/model-serving-api-6688cd56bd-qg44q: AddedInterface - Add eth0 [10.129.3.201/23] from ovn-kubernetes 2026/07/10 08:12:37 08:08:45 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/10 08:12:37 08:08:45 INFO Pod/model-registry-operator-controller-manager-7f75765c5b-tct75: AddedInterface - Add eth0 [10.129.3.200/23] from ovn-kubernetes 2026/07/10 08:12:37 08:08:45 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/10 08:12:37 08:08:45 INFO Pod/odh-model-controller-867855946f-jv4wn: AddedInterface - Add eth0 [10.129.3.199/23] from ovn-kubernetes 2026/07/10 08:12:37 08:08:45 INFO Pod/odh-model-controller-867855946f-jv4wn: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/10 08:12:37 08:08:45 INFO Pod/model-registry-operator-controller-manager-7f75765c5b-tct75: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-registry-operator-rhel9@sha256:ebcec5c43468283ed3cbab50530f939166f8f3e8c9d306e91a8107d05d7fa9d4" 2026/07/10 08:12:37 08:08:44 INFO Pod/maas-controller-74bc958949-kf25x: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:521fc9d9f6139ed08f80315acbdc95b68b90b12c3e866c76e9141b3e23e2fc0b" 2026/07/10 08:12:37 08:08:44 WARN Pod/model-registry-operator-controller-manager-7f75765c5b-tct75: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 2026/07/10 08:12:37 08:08:44 INFO Pod/maas-controller-74bc958949-kf25x: AddedInterface - Add eth0 [10.129.3.198/23] from ovn-kubernetes 2026/07/10 08:12:37 08:08:43 INFO Pod/model-registry-operator-controller-manager-7f75765c5b-tct75: Scheduled - Successfully assigned redhat-ods-applications/model-registry-operator-controller-manager-7f75765c5b-tct75 to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/10 08:12:37 08:08:43 INFO Pod/odh-model-controller-867855946f-jv4wn: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-867855946f-jv4wn to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/10 08:12:37 08:08:43 INFO Pod/maas-controller-74bc958949-kf25x: Scheduled - Successfully assigned redhat-ods-applications/maas-controller-74bc958949-kf25x to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/10 08:12:37 08:08:43 INFO Deployment/model-registry-operator-controller-manager: ScalingReplicaSet - Scaled up replica set model-registry-operator-controller-manager-7f75765c5b from 0 to 1 2026/07/10 08:12:37 08:08:43 WARN Pod/odh-model-controller-867855946f-jv4wn: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/10 08:12:37 08:08:43 INFO ReplicaSet/odh-model-controller-867855946f: SuccessfulCreate - Created pod: odh-model-controller-867855946f-jv4wn 2026/07/10 08:12:37 08:08:43 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-867855946f from 0 to 1 2026/07/10 08:12:37 08:08:43 INFO Deployment/maas-controller: ScalingReplicaSet - Scaled up replica set maas-controller-74bc958949 from 0 to 1 2026/07/10 08:12:37 08:08:43 INFO ReplicaSet/model-registry-operator-controller-manager-7f75765c5b: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-7f75765c5b-tct75 2026/07/10 08:12:37 08:08:43 INFO ReplicaSet/maas-controller-74bc958949: SuccessfulCreate - Created pod: maas-controller-74bc958949-kf25x 2026/07/10 08:12:37 08:08:31 INFO Pod/spark-operator-webhook-9cb9bdf44-hn522: Killing - Stopping container webhook 2026/07/10 08:12:37 08:08:31 INFO Pod/spark-operator-controller-767899d44f-tjtbd: Killing - Stopping container controller 2026/07/10 08:12:37 08:08:17 INFO Lease/spark-operator-controller-lock: LeaderElection - spark-operator-controller-767899d44f-tjtbd_760f9594-2552-4196-a64c-2b460350f88c became leader 2026/07/10 08:12:37 08:08:15 INFO Pod/spark-operator-webhook-9cb9bdf44-hn522: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-webhook-9cb9bdf44-hn522 to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/10 08:12:37 08:08:15 INFO Pod/spark-operator-webhook-9cb9bdf44-hn522: Started - Started container webhook 2026/07/10 08:12:37 08:08:15 INFO Pod/spark-operator-webhook-9cb9bdf44-hn522: AddedInterface - Add eth0 [10.129.3.197/23] from ovn-kubernetes 2026/07/10 08:12:37 08:08:15 INFO Pod/spark-operator-webhook-9cb9bdf44-hn522: Pulled - Container image "registry.redhat.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/10 08:12:37 08:08:15 INFO Pod/spark-operator-webhook-9cb9bdf44-hn522: Created - Created container: webhook 2026/07/10 08:12:37 08:08:15 INFO Deployment/spark-operator-webhook: ScalingReplicaSet - Scaled up replica set spark-operator-webhook-9cb9bdf44 from 0 to 1 2026/07/10 08:12:37 08:08:15 INFO ReplicaSet/spark-operator-webhook-9cb9bdf44: SuccessfulCreate - Created pod: spark-operator-webhook-9cb9bdf44-hn522 2026/07/10 08:12:37 08:08:14 INFO Pod/spark-operator-webhook-9cb9bdf44-ddfpk: Killing - Stopping container webhook 2026/07/10 08:12:37 08:08:14 INFO Pod/spark-operator-webhook-9cb9bdf44-n2mch: Killing - Stopping container webhook 2026/07/10 08:12:37 08:08:00 INFO Pod/spark-operator-controller-767899d44f-tjtbd: Created - Created container: controller 2026/07/10 08:12:37 08:08:00 INFO Pod/spark-operator-controller-767899d44f-tjtbd: AddedInterface - Add eth0 [10.129.3.196/23] from ovn-kubernetes 2026/07/10 08:12:37 08:08:00 INFO Pod/spark-operator-controller-767899d44f-tjtbd: Pulled - Container image "registry.redhat.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/10 08:12:37 08:08:00 INFO Pod/spark-operator-controller-767899d44f-tjtbd: Started - Started container controller 2026/07/10 08:12:37 08:07:59 INFO Pod/spark-operator-controller-767899d44f-tjtbd: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-controller-767899d44f-tjtbd to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/10 08:12:37 08:07:59 INFO Pod/spark-operator-controller-767899d44f-n6pn9: Killing - Stopping container controller 2026/07/10 08:12:37 08:07:59 INFO Pod/spark-operator-controller-767899d44f-nsk87: Killing - Stopping container controller 2026/07/10 08:12:37 08:07:59 INFO ReplicaSet/spark-operator-controller-767899d44f: SuccessfulCreate - Created pod: spark-operator-controller-767899d44f-tjtbd 2026/07/10 08:12:37 08:07:59 INFO Deployment/spark-operator-controller: ScalingReplicaSet - Scaled up replica set spark-operator-controller-767899d44f from 0 to 1 2026/07/10 08:12:37 08:07:48 WARN Endpoints/scheduled-spark-pi-d98ag17s6vus7b145rq0-1783670-2ffed2db-ui-svc: FailedToUpdateEndpoint - Failed to update endpoint redhat-ods-applications/scheduled-spark-pi-d98ag17s6vus7b145rq0-1783670-2ffed2db-ui-svc: Operation cannot be fulfilled on endpoints "scheduled-spark-pi-d98ag17s6vus7b145rq0-1783670-2ffed2db-ui-svc": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/scheduled-spark-pi-d98ag17s6vus7b145rq0-1783670-2ffed2db-ui-svc, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: fcafefb5-117f-474c-9226-c6499586afaa, UID in object meta: 2026/07/10 08:12:37 08:07:39 INFO SparkApplication/scheduled-spark-pi-d98ag17s6vus7b145rq0-1783670818001232240: SparkDriverCompleted - Driver scheduled-spark-pi-d98ag17s6vus7b145rq0-1783670818001232240-driver completed 2026/07/10 08:12:37 08:07:37 INFO SparkApplication/scheduled-spark-pi-d98ag17s6vus7b145rq0-1783670818001232240: SparkExecutorCompleted - Executor spark-pi-d3d6f79f4b111844-exec-1 completed 2026/07/10 08:12:37 08:07:37 INFO Pod/spark-pi-d3d6f79f4b111844-exec-1: Killing - Stopping container spark-kubernetes-executor 2026/07/10 08:12:37 === RESOURCE QUOTAS === 2026/07/10 08:12:37 No resource quotas found in monitored namespaces 2026/07/10 08:12:37 Classification for group 2: infrastructure/storage (code=1006, confidence=medium) 2026/07/10 08:12:37 Evidence: event Pod/workload-variant-autoscaler-controller-manager-7967c6f77c-qlgpp: FailedMount - MountVolume.SetUp failed for volume "epp-metrics-token" : secret "workload-variant-autoscaler-epp-metrics-token" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/workload-variant-autoscaler-controller-manager-7967c6f77c-qlgpp: FailedMount - MountVolume.SetUp failed for volume \"epp-metrics-token\" : secret \"workload-variant-autoscaler-epp-metrics-token\" not found"],"confidence":"medium"} 2026/07/10 08:12:37 Diagnostics complete for failed test: group 2 controller_test.go:753: Stopping: group 2 test failed. 2026/07/10 08:12:37 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/10 08:12:37 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/10 08:16:48 Circuit breaker: success recorded, resetting failure counter (was 1) 2026/07/10 08:25:31 TEST FAILURE DETECTED: group 4 2026/07/10 08:25:31 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 4) === 2026/07/10 08:25:32 === CLUSTER STATE === 2026/07/10 08:25:32 Node: ods-qe-psi-07-b4mqp-master-0 2026/07/10 08:25:32 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 08:25:32 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 08:25:32 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 08:25:32 Condition Ready: True - kubelet is posting ready status 2026/07/10 08:25:32 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/10 08:25:32 Node: ods-qe-psi-07-b4mqp-master-1 2026/07/10 08:25:32 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 08:25:32 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 08:25:32 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 08:25:32 Condition Ready: True - kubelet is posting ready status 2026/07/10 08:25:32 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/10 08:25:32 Node: ods-qe-psi-07-b4mqp-master-2 2026/07/10 08:25:32 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 08:25:32 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 08:25:32 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 08:25:32 Condition Ready: True - kubelet is posting ready status 2026/07/10 08:25:32 Resources: 11500m CPU, 23450732Ki memory allocatable (of 12000m CPU, 24601708Ki memory capacity) 2026/07/10 08:25:32 Node: ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/10 08:25:32 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 08:25:32 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 08:25:32 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 08:25:32 Condition Ready: True - kubelet is posting ready status 2026/07/10 08:25:32 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/10 08:25:32 Node: ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/10 08:25:32 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 08:25:32 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 08:25:32 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 08:25:32 Condition Ready: True - kubelet is posting ready status 2026/07/10 08:25:32 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/10 08:25:32 Node: ods-qe-psi-07-b4mqp-worker-0-jd4pm 2026/07/10 08:25:32 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 08:25:32 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 08:25:32 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 08:25:32 Condition Ready: True - kubelet is posting ready status 2026/07/10 08:25:32 Resources: 11500m CPU, 23452736Ki memory allocatable (of 12000m CPU, 24603712Ki memory capacity) 2026/07/10 08:25:32 Node: ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/10 08:25:32 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 08:25:32 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 08:25:32 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 08:25:32 Condition Ready: True - kubelet is posting ready status 2026/07/10 08:25:32 Resources: 11500m CPU, 23452744Ki memory allocatable (of 12000m CPU, 24603720Ki memory capacity) 2026/07/10 08:25:32 Node: ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/10 08:25:32 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 08:25:32 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 08:25:32 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 08:25:32 Condition Ready: True - kubelet is posting ready status 2026/07/10 08:25:32 Resources: 11500m CPU, 23450720Ki memory allocatable (of 12000m CPU, 24601696Ki memory capacity) 2026/07/10 08:25:32 === DEPLOYMENTS === 2026/07/10 08:25:32 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/10 08:25:32 === PODS === 2026/07/10 08:25:32 Failed to collect pod data: redhat-ods-applications/maas-controller-74bc958949-4gl7z: container manager not ready 2026/07/10 08:25:32 === OPERATOR STATUS === 2026/07/10 08:25:32 Operator deployment: 3/3 ready 2026/07/10 08:25:32 === DSCI STATUS === 2026/07/10 08:25:32 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/10 08:25:32 === DSC STATUS === 2026/07/10 08:25:32 DSC default-dsc: 2026/07/10 08:25:32 Ready: True - 2026/07/10 08:25:32 ProvisioningSucceeded: True - 2026/07/10 08:25:32 ComponentsReady: True - 2026/07/10 08:25:32 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/10 08:25:32 DashboardReady: False - Component ManagementState is set to Removed 2026/07/10 08:25:32 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/10 08:25:32 KserveLLMInferenceServiceDependencies: True - 2026/07/10 08:25:32 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/10 08:25:32 KserveReady: True - 2026/07/10 08:25:32 KueueReady: True - 2026/07/10 08:25:32 LLM-D-WVADependencies: True - 2026/07/10 08:25:32 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/10 08:25:32 ModelControllerReady: True - 2026/07/10 08:25:32 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/10 08:25:32 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/10 08:25:32 OGXReady: False - Component ManagementState is set to Removed 2026/07/10 08:25:32 RayReady: False - Component ManagementState is set to Removed 2026/07/10 08:25:32 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/10 08:25:32 TrainerReady: False - Component ManagementState is set to Removed 2026/07/10 08:25:32 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/10 08:25:32 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/10 08:25:32 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/10 08:25:32 === RECENT EVENTS === 2026/07/10 08:25:32 08:24:22 INFO Pod/maas-controller-74bc958949-4gl7z: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:521fc9d9f6139ed08f80315acbdc95b68b90b12c3e866c76e9141b3e23e2fc0b" in 690ms (690ms including waiting). Image size: 212878878 bytes. 2026/07/10 08:25:32 08:24:21 INFO Pod/maas-controller-74bc958949-4gl7z: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:521fc9d9f6139ed08f80315acbdc95b68b90b12c3e866c76e9141b3e23e2fc0b" 2026/07/10 08:25:32 08:23:29 WARN Pod/maas-controller-74bc958949-4gl7z: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-hjdrc); attempting to pull the image may not succeed. 2026/07/10 08:25:32 08:21:36 INFO Pod/maas-controller-74bc958949-4gl7z: Created - Created container: manager 2026/07/10 08:25:32 08:21:36 INFO Pod/maas-controller-74bc958949-4gl7z: Started - Started container manager 2026/07/10 08:25:32 08:21:36 INFO Pod/maas-controller-74bc958949-4gl7z: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:521fc9d9f6139ed08f80315acbdc95b68b90b12c3e866c76e9141b3e23e2fc0b" in 686ms (686ms including waiting). Image size: 212878878 bytes. 2026/07/10 08:25:32 === RESOURCE QUOTAS === 2026/07/10 08:25:32 No resource quotas found in monitored namespaces 2026/07/10 08:25:32 Classification for group 4: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/10 08:25:32 Evidence: container maas-controller-74bc958949-4gl7z/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-74bc958949-4gl7z_redhat-ods-applications(5b851c81-435e-451f-9f0e-c755797451c1) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-74bc958949-4gl7z/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-74bc958949-4gl7z_redhat-ods-applications(5b851c81-435e-451f-9f0e-c755797451c1)"],"confidence":"medium"} 2026/07/10 08:25:32 Diagnostics complete for failed test: group 4 controller_test.go:753: Stopping: group 4 test failed. 2026/07/10 08:25:32 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/10 08:25:32 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/10 08:26:43 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (1496.42s) === RUN TestOdhOperator 2026/07/10 08:02:37 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/10 08:27:44 TEST FAILURE DETECTED: components 2026/07/10 08:27:44 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/10 08:27:45 === CLUSTER STATE === 2026/07/10 08:27:45 Node: ods-qe-psi-07-b4mqp-master-0 2026/07/10 08:27:45 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 08:27:45 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 08:27:45 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 08:27:45 Condition Ready: True - kubelet is posting ready status 2026/07/10 08:27:45 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/10 08:27:45 Node: ods-qe-psi-07-b4mqp-master-1 2026/07/10 08:27:45 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 08:27:45 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 08:27:45 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 08:27:45 Condition Ready: True - kubelet is posting ready status 2026/07/10 08:27:45 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/10 08:27:45 Node: ods-qe-psi-07-b4mqp-master-2 2026/07/10 08:27:45 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 08:27:45 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 08:27:45 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 08:27:45 Condition Ready: True - kubelet is posting ready status 2026/07/10 08:27:45 Resources: 11500m CPU, 23450732Ki memory allocatable (of 12000m CPU, 24601708Ki memory capacity) 2026/07/10 08:27:45 Node: ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/10 08:27:45 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 08:27:45 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 08:27:45 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 08:27:45 Condition Ready: True - kubelet is posting ready status 2026/07/10 08:27:45 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/10 08:27:45 Node: ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/10 08:27:45 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 08:27:45 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 08:27:45 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 08:27:45 Condition Ready: True - kubelet is posting ready status 2026/07/10 08:27:45 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/10 08:27:45 Node: ods-qe-psi-07-b4mqp-worker-0-jd4pm 2026/07/10 08:27:45 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 08:27:45 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 08:27:45 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 08:27:45 Condition Ready: True - kubelet is posting ready status 2026/07/10 08:27:45 Resources: 11500m CPU, 23452736Ki memory allocatable (of 12000m CPU, 24603712Ki memory capacity) 2026/07/10 08:27:45 Node: ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/10 08:27:45 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 08:27:45 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 08:27:45 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 08:27:45 Condition Ready: True - kubelet is posting ready status 2026/07/10 08:27:45 Resources: 11500m CPU, 23452744Ki memory allocatable (of 12000m CPU, 24603720Ki memory capacity) 2026/07/10 08:27:45 Node: ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/10 08:27:45 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 08:27:45 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 08:27:45 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 08:27:45 Condition Ready: True - kubelet is posting ready status 2026/07/10 08:27:45 Resources: 11500m CPU, 23450720Ki memory allocatable (of 12000m CPU, 24601696Ki memory capacity) 2026/07/10 08:27:45 === DEPLOYMENTS === 2026/07/10 08:27:45 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/10 08:27:45 === PODS === 2026/07/10 08:27:45 Failed to collect pod data: redhat-ods-applications/maas-controller-74bc958949-4gl7z: container manager not ready 2026/07/10 08:27:45 === OPERATOR STATUS === 2026/07/10 08:27:45 Operator deployment: 3/3 ready 2026/07/10 08:27:45 === DSCI STATUS === 2026/07/10 08:27:45 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/10 08:27:45 === DSC STATUS === 2026/07/10 08:27:45 DSC default-dsc: 2026/07/10 08:27:45 Ready: True - 2026/07/10 08:27:45 ProvisioningSucceeded: True - 2026/07/10 08:27:45 ComponentsReady: True - 2026/07/10 08:27:45 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/10 08:27:45 DashboardReady: False - Component ManagementState is set to Removed 2026/07/10 08:27:45 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/10 08:27:45 KserveLLMInferenceServiceDependencies: True - 2026/07/10 08:27:45 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/10 08:27:45 KserveReady: True - 2026/07/10 08:27:45 KueueReady: True - 2026/07/10 08:27:45 LLM-D-WVADependencies: True - 2026/07/10 08:27:45 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/10 08:27:45 ModelControllerReady: True - 2026/07/10 08:27:45 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/10 08:27:45 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/10 08:27:45 OGXReady: False - Component ManagementState is set to Removed 2026/07/10 08:27:45 RayReady: False - Component ManagementState is set to Removed 2026/07/10 08:27:45 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/10 08:27:45 TrainerReady: True - 2026/07/10 08:27:45 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/10 08:27:45 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/10 08:27:45 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/10 08:27:45 === RECENT EVENTS === 2026/07/10 08:27:45 08:27:30 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-57b755898c-c24tp_ed3db26a-bf62-47aa-b439-e5eb55bcb621 became leader 2026/07/10 08:27:45 08:27:27 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-7cd6f695d9-fbrt9_991b67a1-bdea-41a5-837e-15e7eaa0335f became leader 2026/07/10 08:27:45 08:27:13 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-wz5xf_bb8ee854-eaae-4a54-9c9d-54cd583c44a4 became leader 2026/07/10 08:27:45 08:27:10 INFO Pod/llmisvc-controller-manager-57b755898c-c24tp: Started - Started container manager 2026/07/10 08:27:45 08:27:10 INFO Pod/llmisvc-controller-manager-57b755898c-c24tp: AddedInterface - Add eth0 [10.131.0.194/23] from ovn-kubernetes 2026/07/10 08:27:45 08:27:10 INFO Pod/llmisvc-controller-manager-57b755898c-c24tp: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:378244503cc7d4569e676b0bc4b4abb2ce6e3c5f5f879d832464e0e7dc558906" 2026/07/10 08:27:45 08:27:10 INFO Pod/llmisvc-controller-manager-57b755898c-c24tp: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:378244503cc7d4569e676b0bc4b4abb2ce6e3c5f5f879d832464e0e7dc558906" in 781ms (781ms including waiting). Image size: 236366528 bytes. 2026/07/10 08:27:45 08:27:10 INFO Pod/llmisvc-controller-manager-57b755898c-c24tp: Created - Created container: manager 2026/07/10 08:27:45 08:27:09 INFO Pod/kserve-controller-manager-7cd6f695d9-fbrt9: Started - Started container manager 2026/07/10 08:27:45 08:27:09 INFO Pod/kserve-controller-manager-7cd6f695d9-fbrt9: Created - Created container: manager 2026/07/10 08:27:45 08:27:09 WARN Pod/llmisvc-controller-manager-57b755898c-c24tp: FailedMount - MountVolume.SetUp failed for volume "cert" : failed to sync secret [REDACTED] timed out waiting for the condition 2026/07/10 08:27:45 08:27:09 INFO Pod/kserve-controller-manager-7cd6f695d9-fbrt9: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:faa11de719b196594cea0673067e7f28fa3e330ad5181c0abe0e899e585fb009" in 761ms (761ms including waiting). Image size: 238435545 bytes. 2026/07/10 08:27:45 08:27:08 INFO Pod/kserve-controller-manager-7cd6f695d9-fbrt9: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:faa11de719b196594cea0673067e7f28fa3e330ad5181c0abe0e899e585fb009" 2026/07/10 08:27:45 08:27:08 INFO Pod/kserve-controller-manager-7cd6f695d9-fbrt9: AddedInterface - Add eth0 [10.130.2.64/23] from ovn-kubernetes 2026/07/10 08:27:45 08:27:07 INFO Pod/llmisvc-controller-manager-57b755898c-c24tp: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-57b755898c-c24tp to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/10 08:27:45 08:27:07 INFO Pod/kserve-controller-manager-7cd6f695d9-fbrt9: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-7cd6f695d9-fbrt9 to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/10 08:27:45 08:27:07 INFO ReplicaSet/llmisvc-controller-manager-57b755898c: SuccessfulCreate - Created pod: llmisvc-controller-manager-57b755898c-c24tp 2026/07/10 08:27:45 08:27:07 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-57b755898c from 0 to 1 2026/07/10 08:27:45 08:27:07 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7cd6f695d9 from 0 to 1 2026/07/10 08:27:45 08:27:07 INFO ReplicaSet/kserve-controller-manager-7cd6f695d9: SuccessfulCreate - Created pod: kserve-controller-manager-7cd6f695d9-fbrt9 2026/07/10 08:27:45 08:26:57 INFO Pod/model-serving-api-6688cd56bd-m6fvq: Created - Created container: server 2026/07/10 08:27:45 08:26:57 INFO Pod/model-serving-api-6688cd56bd-m6fvq: AddedInterface - Add eth0 [10.131.0.193/23] from ovn-kubernetes 2026/07/10 08:27:45 08:26:57 INFO Pod/odh-model-controller-6849fc49d8-wz5xf: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 636ms (636ms including waiting). Image size: 238959091 bytes. 2026/07/10 08:27:45 08:26:57 INFO Pod/model-serving-api-6688cd56bd-m6fvq: Started - Started container server 2026/07/10 08:27:45 08:26:57 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 08:27:45 08:26:57 INFO Pod/model-serving-api-6688cd56bd-m6fvq: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/10 08:27:45 08:26:57 INFO Pod/odh-model-controller-6849fc49d8-wz5xf: Started - Started container manager 2026/07/10 08:27:45 08:26:57 INFO Pod/odh-model-controller-6849fc49d8-wz5xf: Created - Created container: manager 2026/07/10 08:27:45 08:26:56 INFO Pod/model-serving-api-6688cd56bd-m6fvq: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-m6fvq to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/10 08:27:45 08:26:56 INFO Pod/odh-model-controller-6849fc49d8-wz5xf: AddedInterface - Add eth0 [10.130.2.63/23] from ovn-kubernetes 2026/07/10 08:27:45 08:26:56 INFO Pod/odh-model-controller-6849fc49d8-wz5xf: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/10 08:27:45 08:26:56 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/10 08:27:45 08:26:56 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/10 08:27:45 08:26:56 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-m6fvq 2026/07/10 08:27:45 08:26:55 WARN Pod/odh-model-controller-6849fc49d8-wz5xf: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/10 08:27:45 08:26:54 INFO Pod/odh-model-controller-6849fc49d8-wz5xf: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-wz5xf to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/10 08:27:45 08:26:54 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-wz5xf 2026/07/10 08:27:45 08:26:54 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/10 08:27:45 08:26:44 INFO Pod/llmisvc-controller-manager-57b755898c-lj7dd: Killing - Stopping container manager 2026/07/10 08:27:45 08:26:44 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 08:27:45 08:26:44 INFO Pod/odh-model-controller-6849fc49d8-24m6s: Killing - Stopping container manager 2026/07/10 08:27:45 08:26:44 INFO Pod/model-serving-api-6688cd56bd-m76ss: Killing - Stopping container server 2026/07/10 08:27:45 08:26:44 INFO Pod/kserve-controller-manager-7cd6f695d9-t6gbl: Killing - Stopping container manager 2026/07/10 08:27:45 08:26:23 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-57b755898c-lj7dd_477fdb25-578b-47ea-89f4-ce400961a5ab became leader 2026/07/10 08:27:45 08:26:22 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-7cd6f695d9-t6gbl_2e9a8fee-ee8d-40bb-ad66-1e1fd5d7f0cc became leader 2026/07/10 08:27:45 08:26:07 INFO Pod/llmisvc-controller-manager-57b755898c-lj7dd: Started - Started container manager 2026/07/10 08:27:45 08:26:07 INFO Pod/llmisvc-controller-manager-57b755898c-lj7dd: Created - Created container: manager 2026/07/10 08:27:45 08:26:06 INFO Pod/llmisvc-controller-manager-57b755898c-lj7dd: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:378244503cc7d4569e676b0bc4b4abb2ce6e3c5f5f879d832464e0e7dc558906" in 1.972s (1.972s including waiting). Image size: 236366528 bytes. 2026/07/10 08:27:45 08:26:06 INFO Pod/kserve-controller-manager-7cd6f695d9-t6gbl: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:faa11de719b196594cea0673067e7f28fa3e330ad5181c0abe0e899e585fb009" in 1.028s (1.028s including waiting). Image size: 238435545 bytes. 2026/07/10 08:27:45 08:26:06 INFO Pod/kserve-controller-manager-7cd6f695d9-t6gbl: Created - Created container: manager 2026/07/10 08:27:45 08:26:06 INFO Pod/kserve-controller-manager-7cd6f695d9-t6gbl: Started - Started container manager 2026/07/10 08:27:45 08:26:05 INFO Pod/llmisvc-controller-manager-57b755898c-lj7dd: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:378244503cc7d4569e676b0bc4b4abb2ce6e3c5f5f879d832464e0e7dc558906" 2026/07/10 08:27:45 08:26:05 INFO Pod/kserve-controller-manager-7cd6f695d9-t6gbl: AddedInterface - Add eth0 [10.129.3.208/23] from ovn-kubernetes 2026/07/10 08:27:45 08:26:05 INFO Pod/kserve-controller-manager-7cd6f695d9-t6gbl: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:faa11de719b196594cea0673067e7f28fa3e330ad5181c0abe0e899e585fb009" 2026/07/10 08:27:45 08:26:04 INFO Pod/llmisvc-controller-manager-57b755898c-lj7dd: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-57b755898c-lj7dd to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/10 08:27:45 08:26:04 INFO Pod/kserve-controller-manager-7cd6f695d9-t6gbl: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-7cd6f695d9-t6gbl to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/10 08:27:45 08:26:04 WARN Pod/kserve-controller-manager-7cd6f695d9-t6gbl: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 2026/07/10 08:27:45 08:26:04 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-57b755898c from 0 to 1 2026/07/10 08:27:45 08:26:04 INFO ReplicaSet/kserve-controller-manager-7cd6f695d9: SuccessfulCreate - Created pod: kserve-controller-manager-7cd6f695d9-t6gbl 2026/07/10 08:27:45 08:26:04 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7cd6f695d9 from 0 to 1 2026/07/10 08:27:45 08:26:04 INFO Pod/llmisvc-controller-manager-57b755898c-lj7dd: AddedInterface - Add eth0 [10.131.0.192/23] from ovn-kubernetes 2026/07/10 08:27:45 08:26:04 INFO ReplicaSet/llmisvc-controller-manager-57b755898c: SuccessfulCreate - Created pod: llmisvc-controller-manager-57b755898c-lj7dd 2026/07/10 08:27:45 08:26:02 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-24m6s_fcf1d869-7e55-43ed-9dbb-420ede153be3 became leader 2026/07/10 08:27:45 08:25:53 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-7ccfdd66-t5xn7_097220e0-9674-4a34-8fd5-d1edec9e3d4f became leader 2026/07/10 08:27:45 08:25:46 INFO Pod/model-serving-api-6688cd56bd-m76ss: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/10 08:27:45 08:25:46 INFO Pod/model-serving-api-6688cd56bd-m76ss: Started - Started container server 2026/07/10 08:27:45 08:25:46 INFO Pod/model-serving-api-6688cd56bd-m76ss: Created - Created container: server 2026/07/10 08:27:45 08:25:46 INFO Pod/model-serving-api-6688cd56bd-m76ss: AddedInterface - Add eth0 [10.130.2.62/23] from ovn-kubernetes 2026/07/10 08:27:45 08:25:46 INFO Pod/odh-model-controller-6849fc49d8-24m6s: Started - Started container manager 2026/07/10 08:27:45 08:25:46 INFO Pod/odh-model-controller-6849fc49d8-24m6s: Created - Created container: manager 2026/07/10 08:27:45 08:25:45 INFO Pod/model-serving-api-6688cd56bd-m76ss: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-m76ss to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/10 08:27:45 08:25:45 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/10 08:27:45 08:25:45 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-m76ss 2026/07/10 08:27:45 08:25:45 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/10 08:27:45 08:25:45 INFO Pod/odh-model-controller-6849fc49d8-24m6s: AddedInterface - Add eth0 [10.131.0.191/23] from ovn-kubernetes 2026/07/10 08:27:45 08:25:45 INFO Pod/odh-model-controller-6849fc49d8-24m6s: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/10 08:27:45 08:25:45 INFO Pod/odh-model-controller-6849fc49d8-24m6s: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 660ms (660ms including waiting). Image size: 238959091 bytes. 2026/07/10 08:27:45 08:25:44 INFO Pod/odh-model-controller-6849fc49d8-24m6s: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-24m6s to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/10 08:27:45 08:25:44 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-24m6s 2026/07/10 08:27:45 08:25:44 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/10 08:27:45 08:25:44 WARN Pod/odh-model-controller-6849fc49d8-24m6s: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/10 08:27:45 08:25:37 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-t5xn7: AddedInterface - Add eth0 [10.129.3.207/23] from ovn-kubernetes 2026/07/10 08:27:45 08:25:37 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-t5xn7: Pulled - Container image "registry.redhat.io/rhoai/odh-trainer-rhel9@sha256:59ea523e2df9c1e705c755308ac11420682081cc63d54890e2b15fbd7d8ba570" already present on machine 2026/07/10 08:27:45 08:25:37 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-t5xn7: Created - Created container: manager 2026/07/10 08:27:45 08:25:37 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-t5xn7: Started - Started container manager 2026/07/10 08:27:45 08:25:36 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-t5xn7: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-7ccfdd66-t5xn7 to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/10 08:27:45 08:25:36 INFO ReplicaSet/kubeflow-trainer-controller-manager-7ccfdd66: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-7ccfdd66-t5xn7 2026/07/10 08:27:45 08:25:36 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-7ccfdd66 from 0 to 1 2026/07/10 08:27:45 08:25:33 INFO Pod/llmisvc-controller-manager-57b755898c-69flm: Killing - Stopping container manager 2026/07/10 08:27:45 08:25:33 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 08:27:45 08:25:33 WARN Endpoints/llmisvc-controller-manager-service: FailedToUpdateEndpoint - Failed to update endpoint redhat-ods-applications/llmisvc-controller-manager-service: Operation cannot be fulfilled on endpoints "llmisvc-controller-manager-service": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/llmisvc-controller-manager-service, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: e1ea6e39-7996-446e-a888-7e43b1644819, UID in object meta: 2026/07/10 08:27:45 08:25:33 WARN Endpoints/odh-model-controller-metrics-service: FailedToUpdateEndpoint - Failed to update endpoint redhat-ods-applications/odh-model-controller-metrics-service: Operation cannot be fulfilled on endpoints "odh-model-controller-metrics-service": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/odh-model-controller-metrics-service, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 2ca78c23-0ab2-4787-8e24-5b8654976213, UID in object meta: 2026/07/10 08:27:45 08:25:33 INFO Pod/model-serving-api-6688cd56bd-d5stn: Killing - Stopping container server 2026/07/10 08:27:45 08:25:33 INFO Pod/odh-model-controller-6849fc49d8-7sxnx: Killing - Stopping container manager 2026/07/10 08:27:45 08:25:33 INFO Pod/kserve-controller-manager-7cd6f695d9-jvnt8: Killing - Stopping container manager 2026/07/10 08:27:45 08:24:22 INFO Pod/maas-controller-74bc958949-4gl7z: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:521fc9d9f6139ed08f80315acbdc95b68b90b12c3e866c76e9141b3e23e2fc0b" in 690ms (690ms including waiting). Image size: 212878878 bytes. 2026/07/10 08:27:45 08:24:21 INFO Pod/maas-controller-74bc958949-4gl7z: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:521fc9d9f6139ed08f80315acbdc95b68b90b12c3e866c76e9141b3e23e2fc0b" 2026/07/10 08:27:45 08:23:29 WARN Pod/maas-controller-74bc958949-4gl7z: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-hjdrc); attempting to pull the image may not succeed. 2026/07/10 08:27:45 === RESOURCE QUOTAS === 2026/07/10 08:27:45 No resource quotas found in monitored namespaces 2026/07/10 08:27:45 Classification for components: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/10 08:27:45 Evidence: container maas-controller-74bc958949-4gl7z/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-74bc958949-4gl7z_redhat-ods-applications(5b851c81-435e-451f-9f0e-c755797451c1) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-74bc958949-4gl7z/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-74bc958949-4gl7z_redhat-ods-applications(5b851c81-435e-451f-9f0e-c755797451c1)"],"confidence":"medium"} 2026/07/10 08:27:45 Diagnostics complete for failed test: components controller_test.go:484: Stopping: components test failed. 2026/07/10 08:27:45 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/10 08:27:45 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/10 08:27:48 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (1511.11s)