=== RUN TestOdhOperator/DSCInitialization_and_DataScienceCluster_validation_E2E_Tests/Validate_DSCI_webhook_denies_invalid_PEM_on_update testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/DSCInitialization_and_DataScienceCluster_validation_E2E_Tests/Validate_DSCI_webhook_denies_invalid_PEM_on_update (300.01s) === RUN TestOdhOperator/DSCInitialization_and_DataScienceCluster_validation_E2E_Tests 2026-07-09T08:27:19Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.20.14"}} test_context_test.go:1675: Timed out after 300.008s. Webhook should deny update with invalid PEM in customCABundle The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1664 with: Webhook should deny update with invalid PEM in customCABundle Expected an error to have occurred. Got: <nil>: nil --- FAIL: TestOdhOperator/DSCInitialization_and_DataScienceCluster_validation_E2E_Tests (300.23s) === RUN TestOdhOperator/components/group_2/kueue/Validate_autoCreateQueues_disabled_skips_queue_creation helper_test.go:156: Setting Kueue component (default-kueue) to Removed mode to start with a clean state. helper_test.go:156: Cleaning up any existing Kueue test resources. helper_test.go:156: Cleaning up Kueue cluster-scoped resources helper_test.go:156: Will attempt to delete 3 Kueue resources helper_test.go:156: Attempting to delete ClusterQueue /default helper_test.go:156: Successfully processed deletion of ClusterQueue /default helper_test.go:156: Attempting to delete ResourceFlavor /default-flavor helper_test.go:156: Successfully processed deletion of ResourceFlavor /default-flavor helper_test.go:156: Attempting to delete Kueue /cluster helper_test.go:156: Successfully processed deletion of Kueue /cluster helper_test.go:156: Ensuring LWS Operator is installed (namespace=openshift-lws-operator, name=leader-worker-set). helper_test.go:156: Ensuring LeaderWorkerSetOperator CR exists (namespace=openshift-lws-operator, name=cluster). helper_test.go:156: Ensuring OCP Kueue Operator is installed (namespace=openshift-kueue-operator, name=kueue-operator). helper_test.go:156: Creating test namespace (test-kueue-no-queues-d97lsb8fboqc7b5krpmg) with Kueue management annotation. helper_test.go:156: Creating Kueue ConfigMap (kueue-manager-config) to verify default Kueue resource is created correctly. helper_test.go:156: Creating Kueue ConfigMap (kueue-manager-config) in namespace (redhat-ods-applications). helper_test.go:156: Updating DSC to set Kueue to Unmanaged with autoCreateQueues=false (default). 2026-07-09T08:39:19Z INFO unknown field "spec.components.kueue.autoCreateQueues" helper_test.go:156: Verifying Kueue configuration (cluster) is still created. helper_test.go:156: Verifying no ClusterQueue, ResourceFlavor or LocalQueue is created (autoCreateQueues=false). testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_2/kueue/Validate_autoCreateQueues_disabled_skips_queue_creation (361.02s) === RUN TestOdhOperator/components/group_2/kueue === PAUSE TestOdhOperator/components/group_2/kueue === CONT TestOdhOperator/components/group_2/kueue test_context_test.go:1106: cleanup: best-effort delete failed for /, Kind= redhat-ods-applications/kueue-manager-config: Object 'Kind' is missing in 'unstructured object has no kind' test_context_test.go:615: Timed out after 300.008s. The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1824 with: Expected resource list 'default' of kind 'ClusterQueue' to be empty, but it contains resources. Expected <[]unstructured.Unstructured | len:1, cap:1>: [ { Object: { "apiVersion": <string>"kueue.x-k8s.io/v1beta2", "kind": <string>"ClusterQueue", "metadata": <map[string]interface {} | len:9>{ "annotations": <map[string]interface {} | len:5>{ "platform.opendatahub.io/instance.name": <string>"default-kueue", "platform.opendatahub.io/instance.uid": <string>"a6307eb6-c5e8-4746-8991-939816f43044", "platform.opendatahub.io/type": <string>"OpenShift AI Self-Managed", "platform.opendatahub.io/version": <string>"3.5.0-ea.2", "platform.opendatahub.io/instance.generation": <string>"1", }, "creationTimestamp": <string>"2026-07-09T08:39:09Z", "finalizers": <[]interface {} | len:1, cap:1>[ <string>"kueue.x-k8s.io/resource-in-use", ], "generation": <int64>1, "labels": <map[string]interface {} | len:1>{ "platform.opendatahub.io/part-of": <string>"kueue", }, "resourceVersion": <string>"250216350", "managedFields": <[]interface {} | len:3, cap:4>[ <map[string]interface {} | len:6>{ "fieldsV1": <map[string]interface {} | len:2>{ "f:metadata": <map[string]interface {} | len:2>{ "f:labels": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/part-of": <map[string]interface {} | len:0>{}, }, "f:annotations": <map[string]interface {} | len:6>{ ".": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/instance.generation": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/instance.name": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/instance.uid": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/type": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/version": <map[string]interface {} | len:0>{}, }, }, "f:spec": <map[string]interface {} | len:7>{ "f:stopPolicy": <map[string]interface {} | len:0>{}, ".": <map[string]interface {} | len:0>{}, "f:flavorFungibility": <map[string]interface {} | len:3>{ "f:whenCanPreempt": <map[string]interface {} | len:0>{}, ".": <map[string]interface {} | len:0>{}, "f:whenCanBorrow": <map[string]interface {} | len:0>{}, }, "f:namespaceSelector": <map[string]interface {} | len:0>{}, "f:preemption": <map[string]interface {} | len:4>{ ".": <map[string]interface {} | len:0>{}, "f:borrowWithinCohort": <map[string]interface {} | len:2>{...: ..., ...: ...}, "f:reclaimWithinCohort": <map[string]interface {} | len:0>{}, "f:withinClusterQueue": <map[string]interface {} | len:0>{}, }, "f:queueingStrategy": <map[string]interface {} | len:0>{}, "f:resourceGroups": <map[string]interface {} | len:0>{}, }, }, "manager": <string>"manager", "operation": <string>"Update", "time": <string>"2026-07-09T08:39:09Z", "apiVersion": <string>"kueue.x-k8s.io/v1beta2", "fieldsType": <string>"FieldsV1", }, <map[string]interface {} | len:6>{ "operation": <string>"Update", "time": <string>"2026-07-09T08:44:23Z", "apiVersion": <string>"kueue.x-k8s.io/v1beta2", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:1>{ "f:metadata": <map[string]interface {} | len:1>{ "f:finalizers": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "v:\"kueue.x-k8s.io/resource-in-use\"": <map[string]interface {} | len:0>{}, }, }, }, "manager": <string>"kueue", }, <map[string]interface {} | len:7>{ "apiVersion": <string>"kueue.x-k8s.io/v1beta2", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:1>{ "f:status": <map[string]interface {} | len:7>{ "f:reservingWorkloads": <map[string]interface {} | len:0>{}, ".": <map[string]interface {} | len:0>{}, "f:admittedWorkloads": <map[string]interface {} | len:0>{}, "f:conditions": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "k:{\"type\":\"Active\"}": <map[string]interface {} | len:7>{...: ..., ...: ..., ...: ..., ...: ..., ...: ..., ...: ..., ...: ...}, }, "f:flavorsReservation": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "k:{\"name\":\"default-flavor\"}": <map[string]interface {} | len:3>{...: ..., ...: ..., ...: ...}, }, "f:flavorsUsage": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "k:{\"name\":\"default-flavor\"}": <map[string]interface {} | len:3>{...: ..., ...: ..., ...: ...}, }, "f:pendingWorkloads": <map[string]interface {} | len:0>{}, }, }, "manager": <string>"kueue", "operation": <string>"Update", "subresource": <string>"status", "time": <string>"2026-07-09T08:44:23Z", }, ], "name": <string>"default", "uid": <string>"93bed264-cbdc-4bd3-92b7-cdc453aa3216", }, "spec": <map[string]interface {} | len:6>{ "queueingStrategy": <string>"BestEffortFIFO", "resourceGroups": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:2>{ "coveredResources": <[]interface {} | len:2, cap:2>[<string>"cpu", <string>"memory"], "flavors": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:2>{ "name": <string>"default-flavor", "resources": <[]interface {} | len:2, cap:2>[ <map[string]interface {} | len:2>{...: ..., ...: ...}, <map[string]interface {} | len:2>{...: ..., ...: ...}, ], }, ], }, ], "stopPolicy": <string>"None", "flavorFungibility": <map[string]interface {} | len:2>{ "whenCanBorrow": <string>"MayStopSearch", "whenCanPreempt": <string>"TryNextFlavor", }, "namespaceSelector": <map[string]interface {} | len:1>{ "matchLabels": <map[string]interface {} | len:1>{ "kueue.openshift.io/managed": <string>"true", }, }, "preemption": <map[string]interface {} | len:3>{ "borrowWithinCohort": <map[string]interface {} | len:1>{ "policy": <string>"Never", }, "reclaimWithinCohort": <string>"Never", "withinClusterQueue": <string>"Never", }, }, "status": <map[string]interface {} | len:6>{ "admittedWorkloads": <int64>0, "conditions": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:6>{ "status": <string>"True", "type": <string>"Active", "lastTransitionTime": <string>"2026-07-09T08:44:23Z", "message": <string>"Can admit new workloads", "observedGeneration": <int64>1, "reason": <string>"Ready", }, ], "flavorsReservation": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:2>{ "resources": <[]interface {} | len:2, cap:2>[ <map[string]interface {} | len:3>{ "name": <string>"cpu", "total": <string>"0", "borrowed": <string>"0", }, <map[string]interface {} | len:3>{ "borrowed": <string>"0", "name": <string>"memory", "total": <string>"0", }, ], "name": <string>"default-flavor", }, ], "flavorsUsage": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:2>{ "name": <string>"default-flavor", "resources": <[]interface {} | len:2, cap:2>[ <map[string]interface {} | len:3>{ "borrowed": <string>"0", "name": <string>"cpu", "total": <string>"0", }, <map[string]interface {} | len:3>{ "borrowed": <string>"0", "name": <string>"memory", "total": <string>"0", }, ], }, ], "pendingWorkloads": <int64>0, "reservingWorkloads": <int64>0, }, }, }, ] to be empty test_context_test.go:1106: cleanup: best-effort delete failed for /, Kind= redhat-ods-applications/kueue-manager-config: Object 'Kind' is missing in 'unstructured object has no kind' --- FAIL: TestOdhOperator/components/group_2/kueue (423.24s) === RUN TestOdhOperator/components/group_2 --- FAIL: TestOdhOperator/components/group_2 (0.00s) === RUN TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable helper_test.go:156: Verifying Tenant models-as-a-service/default-tenant is present before disable helper_test.go:156: Disabling MaaS subcomponent (setting to Removed) helper_test.go:156: Waiting for Tenant models-as-a-service/default-tenant to be deleted testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable (430.17s) === RUN TestOdhOperator/components/group_4/modelsasservice === PAUSE TestOdhOperator/components/group_4/modelsasservice === CONT TestOdhOperator/components/group_4/modelsasservice 2026-07-09T08:49:19Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.20.14"}} controller_test.go:763: Creating MaaS PostgreSQL instance in namespace redhat-ods-applications controller_test.go:763: MaaS PostgreSQL instance and maas-db-config secret created in namespace redhat-ods-applications controller_test.go:763: Creating MaaS GatewayClass and Gateway: openshift-ingress/maas-default-gateway controller_test.go:763: Using hostname=maas.apps.ods-qe-psi-23.osp.rh-ods.com, certSecret=router-certs-default controller_test.go:763: MaaS Gateway openshift-ingress/maas-default-gateway created with HTTP+HTTPS listeners test_context_test.go:615: Timed out after 420.008s. The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1824 with: Expected resource list 'models-as-a-service/default-tenant' of kind 'Tenant' to be empty, but it contains resources. Expected <[]unstructured.Unstructured | len:1, cap:1>: [ { Object: { "kind": <string>"Tenant", "metadata": <map[string]interface {} | len:8>{ "resourceVersion": <string>"250225880", "uid": <string>"f99c5448-ac52-4505-ab36-b77408d560a9", "creationTimestamp": <string>"2026-07-09T08:50:18Z", "generation": <int64>1, "managedFields": <[]interface {} | len:2, cap:2>[ <map[string]interface {} | len:6>{ "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:2>{ "f:metadata": <map[string]interface {} | len:1>{ "f:ownerReferences": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "k:{\"uid\":\"91c3ff7a-c57b-4b01-bb3e-ffa5f99e6e99\"}": <map[string]interface {} | len:0>{}, }, }, "f:spec": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "f:gatewayRef": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:name": <map[string]interface {} | len:0>{}, "f:namespace": <map[string]interface {} | len:0>{}, }, }, }, "manager": <string>"manager", "operation": <string>"Update", "time": <string>"2026-07-09T08:50:18Z", }, <map[string]interface {} | len:7>{ "time": <string>"2026-07-09T08:50:29Z", "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:1>{ "f:status": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:conditions": <map[string]interface {} | len:0>{}, "f:phase": <map[string]interface {} | len:0>{}, }, }, "manager": <string>"manager", "operation": <string>"Update", "subresource": <string>"status", }, ], "name": <string>"default-tenant", "namespace": <string>"models-as-a-service", "ownerReferences": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:4>{ "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "kind": <string>"Config", "name": <string>"default", "uid": <string>"91c3ff7a-c57b-4b01-bb3e-ffa5f99e6e99", }, ], }, "spec": <map[string]interface {} | len:1>{ "gatewayRef": <map[string]interface {} | len:2>{ "namespace": <string>"openshift-ingress", "name": <string>"maas-default-gateway", }, }, "status": <map[string]interface {} | len:2>{ "conditions": <[]interface {} | len:5, cap:8>[ <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-07-09T08:50:21Z", "message": <string>"AuthConfig CRD (Kuadrant) is available on the cluster", "observedGeneration": <int64>1, "reason": <string>"DependenciesMet", "status": <string>"True", "type": <string>"DependenciesAvailable", }, <map[string]interface {} | len:6>{ "reason": <string>"PrerequisitesMet", "status": <string>"True", "type": <string>"MaaSPrerequisitesAvailable", "lastTransitionTime": <string>"2026-07-09T08:50:21Z", "message": <string>"All prerequisites are satisfied", "observedGeneration": <int64>1, }, <map[string]interface {} | len:6>{ "status": <string>"False", "type": <string>"Degraded", "lastTransitionTime": <string>"2026-07-09T08:50:21Z", "message": <string>"All prerequisites are satisfied", "observedGeneration": <int64>1, "reason": <string>"PrerequisitesMet", }, <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-07-09T08:50:29Z", "message": <string>"maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"DeploymentsReady", "status": <string>"True", "type": <string>"DeploymentsAvailable", }, <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-07-09T08:50:29Z", "message": <string>"MaaS platform manifests applied and maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"Reconciled", "status": <string>"True", "type": <string>"Ready", }, ], "phase": <string>"Active", }, "apiVersion": <string>"maas.opendatahub.io/v1alpha1", }, }, ] to be empty --- FAIL: TestOdhOperator/components/group_4/modelsasservice (501.28s) === RUN TestOdhOperator/components/group_4 --- FAIL: TestOdhOperator/components/group_4 (0.00s) === RUN TestOdhOperator/components 2026/07/09 08:45:10 TEST FAILURE DETECTED: group 2 2026/07/09 08:45:10 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 2) === 2026/07/09 08:45:10 === CLUSTER STATE === 2026/07/09 08:45:10 Node: ods-qe-psi-23-v5877-master-0 2026/07/09 08:45:10 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 08:45:10 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 08:45:10 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 08:45:10 Condition Ready: True - kubelet is posting ready status 2026/07/09 08:45:10 Resources: 11500m CPU, 23450692Ki memory allocatable (of 12000m CPU, 24601668Ki memory capacity) 2026/07/09 08:45:10 Node: ods-qe-psi-23-v5877-master-1 2026/07/09 08:45:10 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 08:45:10 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 08:45:10 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 08:45:10 Condition Ready: True - kubelet is posting ready status 2026/07/09 08:45:10 Resources: 11500m CPU, 23450696Ki memory allocatable (of 12000m CPU, 24601672Ki memory capacity) 2026/07/09 08:45:10 Node: ods-qe-psi-23-v5877-master-2 2026/07/09 08:45:10 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 08:45:10 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 08:45:10 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 08:45:10 Condition Ready: True - kubelet is posting ready status 2026/07/09 08:45:10 Resources: 11500m CPU, 23450696Ki memory allocatable (of 12000m CPU, 24601672Ki memory capacity) 2026/07/09 08:45:10 Node: ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/09 08:45:10 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 08:45:10 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 08:45:10 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 08:45:10 Condition Ready: True - kubelet is posting ready status 2026/07/09 08:45:10 Resources: 11500m CPU, 23450688Ki memory allocatable (of 12000m CPU, 24601664Ki memory capacity) 2026/07/09 08:45:10 Node: ods-qe-psi-23-v5877-worker-0-gdv9r 2026/07/09 08:45:10 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 08:45:10 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 08:45:10 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 08:45:10 Condition Ready: True - kubelet is posting ready status 2026/07/09 08:45:10 Resources: 11500m CPU, 23450696Ki memory allocatable (of 12000m CPU, 24601672Ki memory capacity) 2026/07/09 08:45:10 === DEPLOYMENTS === 2026/07/09 08:45:10 Namespace: kube-system 2026/07/09 08:45:10 Namespace: redhat-ods-applications 2026/07/09 08:45:10 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/09 08:45:10 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/09 08:45:10 === PODS === 2026/07/09 08:45:10 Namespace redhat-ods-applications: no problematic pods found 2026/07/09 08:45:10 Namespace kube-system: no problematic pods found 2026/07/09 08:45:10 === OPERATOR STATUS === 2026/07/09 08:45:10 Operator deployment: 3/3 ready 2026/07/09 08:45:10 === DSCI STATUS === 2026/07/09 08:45:10 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/09 08:45:10 === DSC STATUS === 2026/07/09 08:45:10 Failed to collect DSC data: default-dsc unhealthy conditions: ModelsAsServiceReady=False 2026/07/09 08:45:10 === RECENT EVENTS === 2026/07/09 08:45:10 08:41:52 INFO Pod/maas-api-84449bdd5d-ds2qw: Killing - Stopping container maas-api 2026/07/09 08:45:10 08:41:51 INFO Pod/model-serving-api-6688cd56bd-fbg76: Killing - Stopping container server 2026/07/09 08:45:10 08:41:51 INFO Pod/kserve-controller-manager-6c67dcb69c-762kd: Killing - Stopping container manager 2026/07/09 08:45:10 08:41:51 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-9c7sh: Killing - Stopping container manager 2026/07/09 08:45:10 08:41:51 WARN Pod/odh-model-controller-867855946f-nmb6d: Unhealthy - Readiness probe failed: Get "http://10.128.3.66:8081/readyz": dial tcp 10.128.3.66:8081: connect: connection refused 2026/07/09 08:45:10 08:41:51 INFO Pod/maas-controller-85bdcf9988-xxrwq: Killing - Stopping container manager 2026/07/09 08:45:10 08:41:51 INFO Pod/model-registry-operator-controller-manager-77866f469b-prpr5: Killing - Stopping container manager 2026/07/09 08:45:10 08:41:51 INFO Pod/odh-model-controller-867855946f-nmb6d: Killing - Stopping container manager 2026/07/09 08:45:10 08:41:50 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/09 08:45:10 08:41:50 INFO Pod/llmisvc-controller-manager-5d65df97bb-9rltk: Killing - Stopping container manager 2026/07/09 08:45:10 08:41:38 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-867855946f-nmb6d_ae7b8364-4778-461c-a57f-3047d588687d became leader 2026/07/09 08:45:10 08:41:35 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-9c7sh: Started - Started container manager 2026/07/09 08:45:10 08:41:35 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-5bd688895b-9c7sh_34aa0156-0eb4-4b75-ab13-2cd1ddc917b2 became leader 2026/07/09 08:45:10 08:41:35 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-9c7sh: AddedInterface - Add eth0 [10.131.0.122/23] from ovn-kubernetes 2026/07/09 08:45:10 08:41:35 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-9c7sh: Created - Created container: manager 2026/07/09 08:45:10 08:41:35 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-9c7sh: Pulled - Container image "registry.redhat.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" already present on machine 2026/07/09 08:45:10 08:41:34 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-vzznq: Killing - Stopping container manager 2026/07/09 08:45:10 08:41:34 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-9c7sh: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-5bd688895b-9c7sh to ods-qe-psi-23-v5877-worker-0-gdv9r 2026/07/09 08:45:10 08:41:34 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-5bd688895b-9c7sh 2026/07/09 08:45:10 08:41:34 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-qpmxs: Killing - Stopping container manager 2026/07/09 08:45:10 08:41:34 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-5bd688895b from 0 to 1 2026/07/09 08:45:10 08:41:20 INFO Pod/odh-model-controller-867855946f-nmb6d: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 625ms (625ms including waiting). Image size: 238959091 bytes. 2026/07/09 08:45:10 08:41:20 INFO Pod/odh-model-controller-867855946f-nmb6d: Created - Created container: manager 2026/07/09 08:45:10 08:41:20 INFO Pod/odh-model-controller-867855946f-nmb6d: Started - Started container manager 2026/07/09 08:45:10 08:41:19 INFO Pod/odh-model-controller-867855946f-nmb6d: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/09 08:45:10 08:41:19 INFO Pod/odh-model-controller-867855946f-knlwj: Killing - Stopping container manager 2026/07/09 08:45:10 08:41:19 INFO Pod/odh-model-controller-867855946f-nmb6d: AddedInterface - Add eth0 [10.128.3.66/23] from ovn-kubernetes 2026/07/09 08:45:10 08:41:19 INFO Pod/odh-model-controller-867855946f-t4cgb: Killing - Stopping container manager 2026/07/09 08:45:10 08:41:19 INFO ReplicaSet/odh-model-controller-867855946f: SuccessfulCreate - Created pod: odh-model-controller-867855946f-nmb6d 2026/07/09 08:45:10 08:41:19 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-867855946f from 0 to 1 2026/07/09 08:45:10 08:41:19 INFO Pod/odh-model-controller-867855946f-nmb6d: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-867855946f-nmb6d to ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/09 08:45:10 08:41:04 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-fbg76 2026/07/09 08:45:10 08:41:04 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/09 08:45:10 08:41:04 INFO Pod/model-serving-api-6688cd56bd-tnvp6: Killing - Stopping container server 2026/07/09 08:45:10 08:41:04 INFO Pod/model-serving-api-6688cd56bd-ltjkl: Killing - Stopping container server 2026/07/09 08:45:10 08:41:04 INFO Pod/model-serving-api-6688cd56bd-fbg76: Started - Started container server 2026/07/09 08:45:10 08:41:04 INFO Pod/model-serving-api-6688cd56bd-fbg76: Created - Created container: server 2026/07/09 08:45:10 08:41:04 INFO Pod/model-serving-api-6688cd56bd-fbg76: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/09 08:45:10 08:41:04 INFO Pod/model-serving-api-6688cd56bd-fbg76: AddedInterface - Add eth0 [10.131.0.121/23] from ovn-kubernetes 2026/07/09 08:45:10 08:41:04 INFO Pod/model-serving-api-6688cd56bd-fbg76: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-fbg76 to ods-qe-psi-23-v5877-worker-0-gdv9r 2026/07/09 08:45:10 08:40:54 WARN Pod/odh-model-controller-867855946f-t4cgb: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/09 08:45:10 === RESOURCE QUOTAS === 2026/07/09 08:45:10 No resource quotas found in monitored namespaces 2026/07/09 08:45:10 Classification for group 2: infrastructure/probe-failure (code=1014, confidence=medium) 2026/07/09 08:45:10 Evidence: event Pod/odh-model-controller-867855946f-nmb6d: Unhealthy - Readiness probe failed: Get "http://10.128.3.66:8081/readyz": dial tcp 10.128.3.66:8081: connect: connection refused FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"probe-failure","error_code":1014,"evidence":["event Pod/odh-model-controller-867855946f-nmb6d: Unhealthy - Readiness probe failed: Get \"http://10.128.3.66:8081/readyz\": dial tcp 10.128.3.66:8081: connect: connection refused"],"confidence":"medium"} 2026/07/09 08:45:10 Diagnostics complete for failed test: group 2 controller_test.go:763: Stopping: group 2 test failed. 2026/07/09 08:45:10 Circuit breaker: failure classified as infrastructure/probe-failure (code=1014) 2026/07/09 08:45:10 Circuit breaker: infrastructure/unknown failure recorded (2/3) 2026/07/09 08:57:40 TEST FAILURE DETECTED: group 4 2026/07/09 08:57:40 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 4) === 2026/07/09 08:57:40 === CLUSTER STATE === 2026/07/09 08:57:40 Node: ods-qe-psi-23-v5877-master-0 2026/07/09 08:57:40 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 08:57:40 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 08:57:40 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 08:57:40 Condition Ready: True - kubelet is posting ready status 2026/07/09 08:57:40 Resources: 11500m CPU, 23450692Ki memory allocatable (of 12000m CPU, 24601668Ki memory capacity) 2026/07/09 08:57:40 Node: ods-qe-psi-23-v5877-master-1 2026/07/09 08:57:40 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 08:57:40 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 08:57:40 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 08:57:40 Condition Ready: True - kubelet is posting ready status 2026/07/09 08:57:40 Resources: 11500m CPU, 23450696Ki memory allocatable (of 12000m CPU, 24601672Ki memory capacity) 2026/07/09 08:57:40 Node: ods-qe-psi-23-v5877-master-2 2026/07/09 08:57:40 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 08:57:40 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 08:57:40 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 08:57:40 Condition Ready: True - kubelet is posting ready status 2026/07/09 08:57:40 Resources: 11500m CPU, 23450696Ki memory allocatable (of 12000m CPU, 24601672Ki memory capacity) 2026/07/09 08:57:40 Node: ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/09 08:57:40 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 08:57:40 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 08:57:40 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 08:57:40 Condition Ready: True - kubelet is posting ready status 2026/07/09 08:57:40 Resources: 11500m CPU, 23450688Ki memory allocatable (of 12000m CPU, 24601664Ki memory capacity) 2026/07/09 08:57:40 Node: ods-qe-psi-23-v5877-worker-0-gdv9r 2026/07/09 08:57:40 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 08:57:40 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 08:57:40 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 08:57:40 Condition Ready: True - kubelet is posting ready status 2026/07/09 08:57:40 Resources: 11500m CPU, 23450696Ki memory allocatable (of 12000m CPU, 24601672Ki memory capacity) 2026/07/09 08:57:40 === DEPLOYMENTS === 2026/07/09 08:57:40 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/09 08:57:40 === PODS === 2026/07/09 08:57:40 Failed to collect pod data: redhat-ods-applications/maas-controller-85bdcf9988-dxgbd: container manager not ready 2026/07/09 08:57:40 === OPERATOR STATUS === 2026/07/09 08:57:40 Operator deployment: 3/3 ready 2026/07/09 08:57:40 === DSCI STATUS === 2026/07/09 08:57:40 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/09 08:57:40 === DSC STATUS === 2026/07/09 08:57:40 DSC default-dsc: 2026/07/09 08:57:40 Ready: True - 2026/07/09 08:57:40 ProvisioningSucceeded: True - 2026/07/09 08:57:40 ComponentsReady: True - 2026/07/09 08:57:40 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/09 08:57:40 DashboardReady: False - Component ManagementState is set to Removed 2026/07/09 08:57:40 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 08:57:40 KserveLLMInferenceServiceDependencies: True - 2026/07/09 08:57:40 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/09 08:57:40 KserveReady: True - 2026/07/09 08:57:40 KueueReady: True - 2026/07/09 08:57:40 LLM-D-WVADependencies: True - 2026/07/09 08:57:40 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 08:57:40 ModelControllerReady: True - 2026/07/09 08:57:40 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/09 08:57:40 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/09 08:57:40 OGXReady: False - Component ManagementState is set to Removed 2026/07/09 08:57:40 RayReady: False - Component ManagementState is set to Removed 2026/07/09 08:57:40 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 08:57:40 TrainerReady: False - Component ManagementState is set to Removed 2026/07/09 08:57:40 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 08:57:40 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/09 08:57:40 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/09 08:57:40 === RECENT EVENTS === 2026/07/09 08:57:40 08:56:37 INFO Pod/maas-controller-85bdcf9988-dxgbd: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 551ms (551ms including waiting). Image size: 212866077 bytes. 2026/07/09 08:57:40 08:56:36 INFO Pod/maas-controller-85bdcf9988-dxgbd: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/09 08:57:40 08:55:50 WARN Pod/maas-controller-85bdcf9988-dxgbd: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-74sj2); attempting to pull the image may not succeed. 2026/07/09 08:57:40 08:53:49 INFO Pod/maas-controller-85bdcf9988-dxgbd: Created - Created container: manager 2026/07/09 08:57:40 08:53:49 INFO Pod/maas-controller-85bdcf9988-dxgbd: Started - Started container manager 2026/07/09 08:57:40 08:53:48 INFO Pod/maas-controller-85bdcf9988-dxgbd: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 506ms (506ms including waiting). Image size: 212866077 bytes. 2026/07/09 08:57:40 === RESOURCE QUOTAS === 2026/07/09 08:57:40 No resource quotas found in monitored namespaces 2026/07/09 08:57:40 Classification for group 4: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/09 08:57:40 Evidence: container maas-controller-85bdcf9988-dxgbd/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-dxgbd_redhat-ods-applications(19aaee35-46ce-4cc0-bc5c-b0df0b026fef) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-85bdcf9988-dxgbd/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-dxgbd_redhat-ods-applications(19aaee35-46ce-4cc0-bc5c-b0df0b026fef)"],"confidence":"medium"} 2026/07/09 08:57:40 Diagnostics complete for failed test: group 4 controller_test.go:763: Stopping: group 4 test failed. 2026/07/09 08:57:40 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/09 08:57:40 Circuit breaker: infrastructure/unknown failure recorded (3/3) 2026/07/09 08:57:40 Circuit breaker: threshold reached (3 failures), running confirming health check... 2026/07/09 08:57:40 Circuit breaker: confirming health check passed — cluster appears healthy, resetting counter --- FAIL: TestOdhOperator/components (1652.04s) === RUN TestOdhOperator 2026/07/09 08:27:18 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/09 08:32:29 TEST FAILURE DETECTED: DSCInitialization and DataScienceCluster validation E2E Tests 2026/07/09 08:32:29 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:DSCInitialization and DataScienceCluster validation E2E Tests) === 2026/07/09 08:32:30 === CLUSTER STATE === 2026/07/09 08:32:30 Node: ods-qe-psi-23-v5877-master-0 2026/07/09 08:32:30 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 08:32:30 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 08:32:30 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 08:32:30 Condition Ready: True - kubelet is posting ready status 2026/07/09 08:32:30 Resources: 11500m CPU, 23450692Ki memory allocatable (of 12000m CPU, 24601668Ki memory capacity) 2026/07/09 08:32:30 Node: ods-qe-psi-23-v5877-master-1 2026/07/09 08:32:30 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 08:32:30 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 08:32:30 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 08:32:30 Condition Ready: True - kubelet is posting ready status 2026/07/09 08:32:30 Resources: 11500m CPU, 23450696Ki memory allocatable (of 12000m CPU, 24601672Ki memory capacity) 2026/07/09 08:32:30 Node: ods-qe-psi-23-v5877-master-2 2026/07/09 08:32:30 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 08:32:30 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 08:32:30 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 08:32:30 Condition Ready: True - kubelet is posting ready status 2026/07/09 08:32:30 Resources: 11500m CPU, 23450696Ki memory allocatable (of 12000m CPU, 24601672Ki memory capacity) 2026/07/09 08:32:30 Node: ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/09 08:32:30 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 08:32:30 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 08:32:30 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 08:32:30 Condition Ready: True - kubelet is posting ready status 2026/07/09 08:32:30 Resources: 11500m CPU, 23450688Ki memory allocatable (of 12000m CPU, 24601664Ki memory capacity) 2026/07/09 08:32:30 Node: ods-qe-psi-23-v5877-worker-0-gdv9r 2026/07/09 08:32:30 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 08:32:30 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 08:32:30 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 08:32:30 Condition Ready: True - kubelet is posting ready status 2026/07/09 08:32:30 Resources: 11500m CPU, 23450696Ki memory allocatable (of 12000m CPU, 24601672Ki memory capacity) 2026/07/09 08:32:30 === DEPLOYMENTS === 2026/07/09 08:32:30 Namespace: redhat-ods-applications 2026/07/09 08:32:30 Deployment dashboard-redirect: 2/2 (READY) 2026/07/09 08:32:30 Deployment data-science-pipelines-operator-controller-manager: 1/1 (READY) 2026/07/09 08:32:30 Deployment feast-operator-controller-manager: 1/1 (READY) 2026/07/09 08:32:30 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/09 08:32:30 Deployment kubeflow-trainer-controller-manager: 1/1 (READY) 2026/07/09 08:32:30 Deployment kubeflow-training-operator: 1/1 (READY) 2026/07/09 08:32:30 Deployment kuberay-operator: 1/1 (READY) 2026/07/09 08:32:30 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/09 08:32:30 Deployment maas-api: 1/1 (READY) 2026/07/09 08:32:30 Deployment maas-controller: 1/1 (READY) 2026/07/09 08:32:30 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/09 08:32:30 Deployment model-registry-operator-controller-manager: 1/1 (READY) 2026/07/09 08:32:30 Deployment model-serving-api: 1/1 (READY) 2026/07/09 08:32:30 Deployment notebook-controller-deployment: 1/1 (READY) 2026/07/09 08:32:30 Deployment odh-model-controller: 1/1 (READY) 2026/07/09 08:32:30 Deployment odh-notebook-controller-manager: 1/1 (READY) 2026/07/09 08:32:30 Deployment ogx-k8s-operator-controller-manager: 1/1 (READY) 2026/07/09 08:32:30 Deployment rhods-dashboard: 2/2 (READY) 2026/07/09 08:32:30 Deployment spark-operator-controller: 1/1 (READY) 2026/07/09 08:32:30 Deployment spark-operator-webhook: 1/1 (READY) 2026/07/09 08:32:30 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/09 08:32:30 Namespace: kube-system 2026/07/09 08:32:30 === PODS === 2026/07/09 08:32:30 Failed to collect pod data: redhat-ods-applications/feast-operator-controller-manager-599ccb9568-7nf5m: container manager restarts=1 2026/07/09 08:32:30 === OPERATOR STATUS === 2026/07/09 08:32:30 Operator deployment: 3/3 ready 2026/07/09 08:32:30 === DSCI STATUS === 2026/07/09 08:32:30 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/09 08:32:30 === DSC STATUS === 2026/07/09 08:32:30 DSC default-dsc: 2026/07/09 08:32:30 Ready: True - 2026/07/09 08:32:30 ProvisioningSucceeded: True - 2026/07/09 08:32:30 ModulesReady: True - All registered modules have ManagementState Removed or are not configured 2026/07/09 08:32:30 AIPipelinesReady: True - 2026/07/09 08:32:30 ComponentsReady: True - 2026/07/09 08:32:30 DashboardReady: True - 2026/07/09 08:32:30 FeastOperatorReady: True - 2026/07/09 08:32:30 KserveLLMInferenceServiceDependencies: True - 2026/07/09 08:32:30 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/09 08:32:30 KserveReady: True - 2026/07/09 08:32:30 KueueReady: True - 2026/07/09 08:32:30 LLM-D-WVADependencies: True - 2026/07/09 08:32:30 MLflowOperatorReady: True - 2026/07/09 08:32:30 MaaSPrerequisitesAvailable: True - MaaS gateway annotations and Authorino TLS are correctly configured 2026/07/09 08:32:30 ModelControllerReady: True - 2026/07/09 08:32:30 ModelRegistryReady: True - 2026/07/09 08:32:30 ModelsAsServiceReady: True - MaaS platform manifests applied and maas-api deployment is available 2026/07/09 08:32:30 OGXReady: True - 2026/07/09 08:32:30 RayReady: True - 2026/07/09 08:32:30 SparkOperatorReady: True - 2026/07/09 08:32:30 TrainerReady: True - 2026/07/09 08:32:30 TrainingOperatorReady: True - 2026/07/09 08:32:30 TrustyAIReady: True - 2026/07/09 08:32:30 WorkbenchesReady: True - 2026/07/09 08:32:30 === RECENT EVENTS === 2026/07/09 08:32:30 08:30:04 INFO Job/maas-api-key-cleanup-29726430: Completed - Job completed 2026/07/09 08:32:30 08:30:04 INFO CronJob/maas-api-key-cleanup: SuccessfulDelete - Deleted job maas-api-key-cleanup-29726385 2026/07/09 08:32:30 08:30:04 INFO CronJob/maas-api-key-cleanup: SawCompletedJob - Saw completed job: maas-api-key-cleanup-29726430, condition: Complete 2026/07/09 08:32:30 08:30:01 INFO Pod/maas-api-key-cleanup-29726430-wpczp: Created - Created container: cleanup 2026/07/09 08:32:30 08:30:01 INFO Pod/maas-api-key-cleanup-29726430-wpczp: Started - Started container cleanup 2026/07/09 08:32:30 08:30:00 INFO Pod/maas-api-key-cleanup-29726430-wpczp: Scheduled - Successfully assigned redhat-ods-applications/maas-api-key-cleanup-29726430-wpczp to ods-qe-psi-23-v5877-worker-0-gdv9r 2026/07/09 08:32:30 08:30:00 INFO Pod/maas-api-key-cleanup-29726430-wpczp: AddedInterface - Add eth0 [10.131.0.75/23] from ovn-kubernetes 2026/07/09 08:32:30 08:30:00 INFO Pod/maas-api-key-cleanup-29726430-wpczp: Pulled - Container image "registry.redhat.io/ubi9/ubi-minimal@sha256:907b68736aa798b2d38255b7aa070b2a70acb90803864a40f05d0ec47556ddd0" already present on machine 2026/07/09 08:32:30 08:30:00 INFO Job/maas-api-key-cleanup-29726430: SuccessfulCreate - Created pod: maas-api-key-cleanup-29726430-wpczp 2026/07/09 08:32:30 08:30:00 INFO CronJob/maas-api-key-cleanup: SuccessfulCreate - Created job maas-api-key-cleanup-29726430 2026/07/09 08:32:30 === RESOURCE QUOTAS === 2026/07/09 08:32:30 No resource quotas found in monitored namespaces 2026/07/09 08:32:30 Classification for DSCInitialization and DataScienceCluster validation E2E Tests: infrastructure/dsci-unhealthy (code=1009, confidence=medium) 2026/07/09 08:32:30 Evidence: DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"dsci-unhealthy","error_code":1009,"evidence":["DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False"],"confidence":"medium"} 2026/07/09 08:32:30 Diagnostics complete for failed test: DSCInitialization and DataScienceCluster validation E2E Tests controller_test.go:478: Stopping: DSCInitialization and DataScienceCluster validation E2E Tests test failed. 2026/07/09 08:32:30 Circuit breaker: failure classified as infrastructure/dsci-unhealthy (code=1009) 2026/07/09 08:32:30 Circuit breaker: infrastructure/unknown failure recorded (1/3) 2026/07/09 09:00:02 TEST FAILURE DETECTED: components 2026/07/09 09:00:02 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/09 09:00:03 === CLUSTER STATE === 2026/07/09 09:00:03 Node: ods-qe-psi-23-v5877-master-0 2026/07/09 09:00:03 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 09:00:03 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 09:00:03 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 09:00:03 Condition Ready: True - kubelet is posting ready status 2026/07/09 09:00:03 Resources: 11500m CPU, 23450692Ki memory allocatable (of 12000m CPU, 24601668Ki memory capacity) 2026/07/09 09:00:03 Node: ods-qe-psi-23-v5877-master-1 2026/07/09 09:00:03 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 09:00:03 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 09:00:03 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 09:00:03 Condition Ready: True - kubelet is posting ready status 2026/07/09 09:00:03 Resources: 11500m CPU, 23450696Ki memory allocatable (of 12000m CPU, 24601672Ki memory capacity) 2026/07/09 09:00:03 Node: ods-qe-psi-23-v5877-master-2 2026/07/09 09:00:03 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 09:00:03 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 09:00:03 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 09:00:03 Condition Ready: True - kubelet is posting ready status 2026/07/09 09:00:03 Resources: 11500m CPU, 23450696Ki memory allocatable (of 12000m CPU, 24601672Ki memory capacity) 2026/07/09 09:00:03 Node: ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/09 09:00:03 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 09:00:03 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 09:00:03 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 09:00:03 Condition Ready: True - kubelet is posting ready status 2026/07/09 09:00:03 Resources: 11500m CPU, 23450688Ki memory allocatable (of 12000m CPU, 24601664Ki memory capacity) 2026/07/09 09:00:03 Node: ods-qe-psi-23-v5877-worker-0-gdv9r 2026/07/09 09:00:03 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 09:00:03 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 09:00:03 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 09:00:03 Condition Ready: True - kubelet is posting ready status 2026/07/09 09:00:03 Resources: 11500m CPU, 23450696Ki memory allocatable (of 12000m CPU, 24601672Ki memory capacity) 2026/07/09 09:00:03 === DEPLOYMENTS === 2026/07/09 09:00:03 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/09 09:00:03 === PODS === 2026/07/09 09:00:03 Failed to collect pod data: redhat-ods-applications/maas-api-key-cleanup-29726460-jbcbr: container cleanup not ready; redhat-ods-applications/maas-controller-85bdcf9988-dxgbd: container manager not ready 2026/07/09 09:00:03 === OPERATOR STATUS === 2026/07/09 09:00:03 Operator deployment: 3/3 ready 2026/07/09 09:00:03 === DSCI STATUS === 2026/07/09 09:00:03 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/09 09:00:03 === DSC STATUS === 2026/07/09 09:00:03 DSC default-dsc: 2026/07/09 09:00:03 Ready: True - 2026/07/09 09:00:03 ProvisioningSucceeded: True - 2026/07/09 09:00:03 ComponentsReady: True - 2026/07/09 09:00:03 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/09 09:00:03 DashboardReady: False - Component ManagementState is set to Removed 2026/07/09 09:00:03 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 09:00:03 KserveLLMInferenceServiceDependencies: True - 2026/07/09 09:00:03 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/09 09:00:03 KserveReady: True - 2026/07/09 09:00:03 KueueReady: True - 2026/07/09 09:00:03 LLM-D-WVADependencies: True - 2026/07/09 09:00:03 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 09:00:03 ModelControllerReady: True - 2026/07/09 09:00:03 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/09 09:00:03 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/09 09:00:03 OGXReady: False - Component ManagementState is set to Removed 2026/07/09 09:00:03 RayReady: False - Component ManagementState is set to Removed 2026/07/09 09:00:03 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 09:00:03 TrainerReady: True - 2026/07/09 09:00:03 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 09:00:03 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/09 09:00:03 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/09 09:00:03 === RECENT EVENTS === 2026/07/09 09:00:03 09:00:01 INFO Pod/maas-api-key-cleanup-29726460-jbcbr: Created - Created container: cleanup 2026/07/09 09:00:03 09:00:01 INFO Pod/maas-api-key-cleanup-29726460-jbcbr: Started - Started container cleanup 2026/07/09 09:00:03 09:00:00 INFO CronJob/maas-api-key-cleanup: SuccessfulCreate - Created job maas-api-key-cleanup-29726460 2026/07/09 09:00:03 09:00:00 INFO Pod/maas-api-key-cleanup-29726460-jbcbr: Pulled - Container image "registry.redhat.io/ubi9/ubi-minimal@sha256:907b68736aa798b2d38255b7aa070b2a70acb90803864a40f05d0ec47556ddd0" already present on machine 2026/07/09 09:00:03 09:00:00 INFO Pod/maas-api-key-cleanup-29726460-jbcbr: Scheduled - Successfully assigned redhat-ods-applications/maas-api-key-cleanup-29726460-jbcbr to ods-qe-psi-23-v5877-worker-0-gdv9r 2026/07/09 09:00:03 09:00:00 INFO Job/maas-api-key-cleanup-29726460: SuccessfulCreate - Created pod: maas-api-key-cleanup-29726460-jbcbr 2026/07/09 09:00:03 09:00:00 INFO Pod/maas-api-key-cleanup-29726460-jbcbr: AddedInterface - Add eth0 [10.131.0.140/23] from ovn-kubernetes 2026/07/09 09:00:03 08:59:37 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-znq8m_b9362ab2-18d5-48fe-83e9-962125e06447 became leader 2026/07/09 09:00:03 08:59:34 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-pcbb9_0c89be32-09f1-4e3d-8ba5-17d7a96ee3a3 became leader 2026/07/09 09:00:03 08:59:22 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-7s28z_4caf958f-08ed-4e87-9705-79710638a71c became leader 2026/07/09 09:00:03 08:59:19 INFO Pod/llmisvc-controller-manager-5d65df97bb-znq8m: Started - Started container manager 2026/07/09 09:00:03 08:59:18 INFO Pod/llmisvc-controller-manager-5d65df97bb-znq8m: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 647ms (647ms including waiting). Image size: 236366599 bytes. 2026/07/09 09:00:03 08:59:18 INFO Pod/kserve-controller-manager-6c67dcb69c-pcbb9: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 521ms (521ms including waiting). Image size: 238435613 bytes. 2026/07/09 09:00:03 08:59:18 INFO Pod/kserve-controller-manager-6c67dcb69c-pcbb9: Created - Created container: manager 2026/07/09 09:00:03 08:59:18 INFO Pod/kserve-controller-manager-6c67dcb69c-pcbb9: Started - Started container manager 2026/07/09 09:00:03 08:59:18 INFO Pod/llmisvc-controller-manager-5d65df97bb-znq8m: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/09 09:00:03 08:59:18 INFO Pod/llmisvc-controller-manager-5d65df97bb-znq8m: AddedInterface - Add eth0 [10.128.3.84/23] from ovn-kubernetes 2026/07/09 09:00:03 08:59:18 INFO Pod/llmisvc-controller-manager-5d65df97bb-znq8m: Created - Created container: manager 2026/07/09 09:00:03 08:59:17 WARN Pod/llmisvc-controller-manager-5d65df97bb-znq8m: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 2026/07/09 09:00:03 08:59:17 INFO Pod/kserve-controller-manager-6c67dcb69c-pcbb9: AddedInterface - Add eth0 [10.128.3.83/23] from ovn-kubernetes 2026/07/09 09:00:03 08:59:17 INFO Pod/kserve-controller-manager-6c67dcb69c-pcbb9: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/09 09:00:03 08:59:16 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-znq8m 2026/07/09 09:00:03 08:59:16 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/09 09:00:03 08:59:16 INFO Pod/kserve-controller-manager-6c67dcb69c-pcbb9: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-pcbb9 to ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/09 09:00:03 08:59:16 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/09 09:00:03 08:59:16 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-pcbb9 2026/07/09 09:00:03 08:59:16 INFO Pod/llmisvc-controller-manager-5d65df97bb-znq8m: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-znq8m to ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/09 09:00:03 08:59:05 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/09 09:00:03 08:59:05 INFO Pod/model-serving-api-6688cd56bd-vmwhw: AddedInterface - Add eth0 [10.128.3.82/23] from ovn-kubernetes 2026/07/09 09:00:03 08:59:05 INFO Pod/model-serving-api-6688cd56bd-vmwhw: Started - Started container server 2026/07/09 09:00:03 08:59:05 INFO Pod/model-serving-api-6688cd56bd-vmwhw: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/09 09:00:03 08:59:05 INFO Pod/model-serving-api-6688cd56bd-vmwhw: Created - Created container: server 2026/07/09 09:00:03 08:59:04 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/09 09:00:03 08:59:04 INFO Pod/odh-model-controller-6849fc49d8-7s28z: Created - Created container: manager 2026/07/09 09:00:03 08:59:04 INFO Pod/odh-model-controller-6849fc49d8-7s28z: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 544ms (544ms including waiting). Image size: 238959091 bytes. 2026/07/09 09:00:03 08:59:04 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/09 09:00:03 08:59:04 INFO Pod/odh-model-controller-6849fc49d8-7s28z: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/09 09:00:03 08:59:04 INFO Pod/model-serving-api-6688cd56bd-vmwhw: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-vmwhw to ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/09 09:00:03 08:59:04 INFO Pod/odh-model-controller-6849fc49d8-7s28z: AddedInterface - Add eth0 [10.128.3.81/23] from ovn-kubernetes 2026/07/09 09:00:03 08:59:04 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-vmwhw 2026/07/09 09:00:03 08:59:04 INFO Pod/odh-model-controller-6849fc49d8-7s28z: Started - Started container manager 2026/07/09 09:00:03 08:59:03 INFO Pod/odh-model-controller-6849fc49d8-7s28z: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-7s28z to ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/09 09:00:03 08:59:03 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/09 09:00:03 08:59:03 WARN Pod/odh-model-controller-6849fc49d8-7s28z: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/09 09:00:03 08:59:03 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-7s28z 2026/07/09 09:00:03 08:58:52 INFO Pod/odh-model-controller-6849fc49d8-69jts: Killing - Stopping container manager 2026/07/09 09:00:03 08:58:52 INFO Pod/model-serving-api-6688cd56bd-wb8l9: Killing - Stopping container server 2026/07/09 09:00:03 08:58:52 INFO Pod/llmisvc-controller-manager-5d65df97bb-p8blt: Killing - Stopping container manager 2026/07/09 09:00:03 08:58:52 INFO Pod/kserve-controller-manager-6c67dcb69c-6r8qb: Killing - Stopping container manager 2026/07/09 09:00:03 08:58:26 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-p8blt_08cac384-0641-4274-8aad-dff62f76e23a became leader 2026/07/09 09:00:03 08:58:24 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-6r8qb_096d6495-9301-4e2f-b9e7-17ef7188bbec became leader 2026/07/09 09:00:03 08:58:12 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-69jts_db9f07d0-7d29-4ef1-8dfe-9fd996788410 became leader 2026/07/09 09:00:03 08:58:08 INFO Pod/llmisvc-controller-manager-5d65df97bb-p8blt: Started - Started container manager 2026/07/09 09:00:03 08:58:08 INFO Pod/llmisvc-controller-manager-5d65df97bb-p8blt: Created - Created container: manager 2026/07/09 09:00:03 08:58:08 INFO Pod/llmisvc-controller-manager-5d65df97bb-p8blt: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 494ms (494ms including waiting). Image size: 236366599 bytes. 2026/07/09 09:00:03 08:58:08 INFO Pod/kserve-controller-manager-6c67dcb69c-6r8qb: Started - Started container manager 2026/07/09 09:00:03 08:58:08 INFO Pod/kserve-controller-manager-6c67dcb69c-6r8qb: Created - Created container: manager 2026/07/09 09:00:03 08:58:08 INFO Pod/kserve-controller-manager-6c67dcb69c-6r8qb: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 609ms (609ms including waiting). Image size: 238435613 bytes. 2026/07/09 09:00:03 08:58:07 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-p8blt 2026/07/09 09:00:03 08:58:07 INFO Pod/llmisvc-controller-manager-5d65df97bb-p8blt: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-p8blt to ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/09 09:00:03 08:58:07 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-6r8qb 2026/07/09 09:00:03 08:58:07 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/09 09:00:03 08:58:07 INFO Pod/kserve-controller-manager-6c67dcb69c-6r8qb: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-6r8qb to ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/09 09:00:03 08:58:07 INFO Pod/kserve-controller-manager-6c67dcb69c-6r8qb: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/09 09:00:03 08:58:07 INFO Pod/kserve-controller-manager-6c67dcb69c-6r8qb: AddedInterface - Add eth0 [10.128.3.79/23] from ovn-kubernetes 2026/07/09 09:00:03 08:58:07 INFO Pod/llmisvc-controller-manager-5d65df97bb-p8blt: AddedInterface - Add eth0 [10.128.3.80/23] from ovn-kubernetes 2026/07/09 09:00:03 08:58:07 INFO Pod/llmisvc-controller-manager-5d65df97bb-p8blt: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/09 09:00:03 08:58:07 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/09 09:00:03 08:58:02 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-7ccfdd66-qvq5l_7db997b8-f095-4d97-beb7-1d014762d713 became leader 2026/07/09 09:00:03 08:57:55 INFO Pod/odh-model-controller-6849fc49d8-69jts: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/09 09:00:03 08:57:55 INFO Pod/odh-model-controller-6849fc49d8-69jts: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 509ms (509ms including waiting). Image size: 238959091 bytes. 2026/07/09 09:00:03 08:57:55 INFO Pod/odh-model-controller-6849fc49d8-69jts: Created - Created container: manager 2026/07/09 09:00:03 08:57:55 INFO Pod/odh-model-controller-6849fc49d8-69jts: Started - Started container manager 2026/07/09 09:00:03 08:57:54 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/09 09:00:03 08:57:54 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/09 09:00:03 08:57:54 INFO Pod/odh-model-controller-6849fc49d8-69jts: AddedInterface - Add eth0 [10.131.0.139/23] from ovn-kubernetes 2026/07/09 09:00:03 08:57:54 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-wb8l9 2026/07/09 09:00:03 08:57:54 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/09 09:00:03 08:57:54 INFO Pod/model-serving-api-6688cd56bd-wb8l9: Started - Started container server 2026/07/09 09:00:03 08:57:54 INFO Pod/model-serving-api-6688cd56bd-wb8l9: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-wb8l9 to ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/09 09:00:03 08:57:54 INFO Pod/model-serving-api-6688cd56bd-wb8l9: AddedInterface - Add eth0 [10.128.3.78/23] from ovn-kubernetes 2026/07/09 09:00:03 08:57:54 INFO Pod/model-serving-api-6688cd56bd-wb8l9: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/09 09:00:03 08:57:54 INFO Pod/model-serving-api-6688cd56bd-wb8l9: Created - Created container: server 2026/07/09 09:00:03 08:57:53 WARN Pod/odh-model-controller-6849fc49d8-69jts: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/09 09:00:03 08:57:52 INFO Pod/odh-model-controller-6849fc49d8-69jts: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-69jts to ods-qe-psi-23-v5877-worker-0-gdv9r 2026/07/09 09:00:03 08:57:52 INFO Pod/llmisvc-controller-manager-5d65df97bb-zt46t: Created - Created container: manager 2026/07/09 09:00:03 08:57:52 INFO Pod/llmisvc-controller-manager-5d65df97bb-zt46t: Killing - Stopping container manager 2026/07/09 09:00:03 08:57:52 WARN Pod/kserve-controller-manager-6c67dcb69c-flq2j: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 2026/07/09 09:00:03 08:57:52 INFO Pod/llmisvc-controller-manager-5d65df97bb-zt46t: Started - Started container manager 2026/07/09 09:00:03 08:57:52 INFO Pod/llmisvc-controller-manager-5d65df97bb-zt46t: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 656ms (656ms including waiting). Image size: 236366599 bytes. 2026/07/09 09:00:03 08:57:52 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-69jts 2026/07/09 09:00:03 08:57:52 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/09 09:00:03 08:57:51 INFO Pod/llmisvc-controller-manager-5d65df97bb-zt46t: AddedInterface - Add eth0 [10.128.3.77/23] from ovn-kubernetes 2026/07/09 09:00:03 08:57:51 INFO Pod/llmisvc-controller-manager-5d65df97bb-zt46t: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/09 09:00:03 08:57:51 INFO Pod/llmisvc-controller-manager-5d65df97bb-zt46t: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-zt46t to ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/09 09:00:03 08:57:51 WARN Pod/llmisvc-controller-manager-5d65df97bb-zt46t: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (llmisvc-controller-manager-dockercfg-hm2lm); attempting to pull the image may not succeed. 2026/07/09 09:00:03 08:57:51 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-zt46t 2026/07/09 09:00:03 08:57:50 WARN Service/llmisvc-controller-manager-service: ClusterIPNotAllocated - Cluster IP [IPv4]: 172.30.106.253 is not allocated; repairing 2026/07/09 09:00:03 08:57:50 INFO Pod/kserve-controller-manager-6c67dcb69c-flq2j: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-flq2j to ods-qe-psi-23-v5877-worker-0-4m76g 2026/07/09 09:00:03 08:57:50 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-flq2j 2026/07/09 09:00:03 08:57:50 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/09 09:00:03 08:57:50 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/09 09:00:03 08:57:46 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-qvq5l: AddedInterface - Add eth0 [10.131.0.138/23] from ovn-kubernetes 2026/07/09 09:00:03 08:57:46 INFO ReplicaSet/kubeflow-trainer-controller-manager-7ccfdd66: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-7ccfdd66-qvq5l 2026/07/09 09:00:03 08:57:46 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-qvq5l: Pulled - Container image "registry.redhat.io/rhoai/odh-trainer-rhel9@sha256:59ea523e2df9c1e705c755308ac11420682081cc63d54890e2b15fbd7d8ba570" already present on machine 2026/07/09 09:00:03 08:57:46 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-qvq5l: Created - Created container: manager 2026/07/09 09:00:03 08:57:46 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-qvq5l: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-7ccfdd66-qvq5l to ods-qe-psi-23-v5877-worker-0-gdv9r 2026/07/09 09:00:03 08:57:46 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-qvq5l: Started - Started container manager 2026/07/09 09:00:03 08:57:46 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-7ccfdd66 from 0 to 1 2026/07/09 09:00:03 08:57:42 INFO Pod/odh-model-controller-6849fc49d8-c2jxh: Killing - Stopping container manager 2026/07/09 09:00:03 08:57:42 INFO Pod/model-serving-api-6688cd56bd-qpgnm: Killing - Stopping container server 2026/07/09 09:00:03 08:57:41 INFO Pod/kserve-controller-manager-6c67dcb69c-zdfks: Killing - Stopping container manager 2026/07/09 09:00:03 08:57:41 INFO Pod/llmisvc-controller-manager-5d65df97bb-67f5j: Killing - Stopping container manager 2026/07/09 09:00:03 08:56:37 INFO Pod/maas-controller-85bdcf9988-dxgbd: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 551ms (551ms including waiting). Image size: 212866077 bytes. 2026/07/09 09:00:03 08:56:36 INFO Pod/maas-controller-85bdcf9988-dxgbd: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/09 09:00:03 08:55:50 WARN Pod/maas-controller-85bdcf9988-dxgbd: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-74sj2); attempting to pull the image may not succeed. 2026/07/09 09:00:03 === RESOURCE QUOTAS === 2026/07/09 09:00:03 No resource quotas found in monitored namespaces 2026/07/09 09:00:03 Classification for components: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/09 09:00:03 Evidence: container maas-controller-85bdcf9988-dxgbd/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-dxgbd_redhat-ods-applications(19aaee35-46ce-4cc0-bc5c-b0df0b026fef) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-85bdcf9988-dxgbd/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-dxgbd_redhat-ods-applications(19aaee35-46ce-4cc0-bc5c-b0df0b026fef)"],"confidence":"medium"} 2026/07/09 09:00:03 Diagnostics complete for failed test: components controller_test.go:491: Stopping: components test failed. 2026/07/09 09:00:03 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/09 09:00:03 Circuit breaker: infrastructure/unknown failure recorded (1/3) Deletion Policy: Never. Skipping deletion tests. 2026/07/09 09:00:04 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (1965.88s)