=== RUN TestOdhOperator/DSCInitialization_and_DataScienceCluster_validation_E2E_Tests/Validate_DSCI_webhook_denies_invalid_PEM_on_update testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/DSCInitialization_and_DataScienceCluster_validation_E2E_Tests/Validate_DSCI_webhook_denies_invalid_PEM_on_update (300.01s) === RUN TestOdhOperator/DSCInitialization_and_DataScienceCluster_validation_E2E_Tests 2026-07-05T17:14:21Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.21.23"}} test_context_test.go:1675: Timed out after 300.009s. Webhook should deny update with invalid PEM in customCABundle The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1664 with: Webhook should deny update with invalid PEM in customCABundle Expected an error to have occurred. Got: <nil>: nil --- FAIL: TestOdhOperator/DSCInitialization_and_DataScienceCluster_validation_E2E_Tests (300.06s) === RUN TestOdhOperator/components/group_2/kueue/Validate_autoCreateQueues_disabled_skips_queue_creation helper_test.go:156: Setting Kueue component (default-kueue) to Removed mode to start with a clean state. helper_test.go:156: Cleaning up any existing Kueue test resources. helper_test.go:156: Cleaning up Kueue cluster-scoped resources helper_test.go:156: Will attempt to delete 3 Kueue resources helper_test.go:156: Attempting to delete ClusterQueue /default helper_test.go:156: Successfully processed deletion of ClusterQueue /default helper_test.go:156: Attempting to delete ResourceFlavor /default-flavor helper_test.go:156: Successfully processed deletion of ResourceFlavor /default-flavor helper_test.go:156: Attempting to delete Kueue /cluster helper_test.go:156: Successfully processed deletion of Kueue /cluster helper_test.go:156: Ensuring LWS Operator is installed (namespace=openshift-lws-operator, name=leader-worker-set). helper_test.go:156: Ensuring LeaderWorkerSetOperator CR exists (namespace=openshift-lws-operator, name=cluster). helper_test.go:156: Ensuring OCP Kueue Operator is installed (namespace=openshift-kueue-operator, name=kueue-operator). helper_test.go:156: Creating test namespace (test-kueue-no-queues-d9597va9rq0s7b72g5m0) with Kueue management annotation. helper_test.go:156: Creating Kueue ConfigMap (kueue-manager-config) to verify default Kueue resource is created correctly. helper_test.go:156: Creating Kueue ConfigMap (kueue-manager-config) in namespace (redhat-ods-applications). helper_test.go:156: Updating DSC to set Kueue to Unmanaged with autoCreateQueues=false (default). 2026-07-05T17:27:35Z INFO unknown field "spec.components.kueue.autoCreateQueues" helper_test.go:156: Verifying Kueue configuration (cluster) is still created. helper_test.go:156: Verifying no ClusterQueue, ResourceFlavor or LocalQueue is created (autoCreateQueues=false). testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_2/kueue/Validate_autoCreateQueues_disabled_skips_queue_creation (320.23s) === RUN TestOdhOperator/components/group_2/kueue === PAUSE TestOdhOperator/components/group_2/kueue === CONT TestOdhOperator/components/group_2/kueue test_context_test.go:1106: cleanup: best-effort delete failed for /, Kind= redhat-ods-applications/kueue-manager-config: Object 'Kind' is missing in 'unstructured object has no kind' test_context_test.go:615: Timed out after 300.004s. The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1824 with: Expected resource list 'default' of kind 'ClusterQueue' to be empty, but it contains resources. Expected <[]unstructured.Unstructured | len:1, cap:1>: [ { Object: { "metadata": <map[string]interface {} | len:8>{ "generation": <int64>1, "labels": <map[string]interface {} | len:1>{ "platform.opendatahub.io/part-of": <string>"kueue", }, "managedFields": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:6>{ "manager": <string>"manager", "operation": <string>"Update", "time": <string>"2026-07-05T17:27:36Z", "apiVersion": <string>"kueue.x-k8s.io/v1beta2", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:2>{ "f:metadata": <map[string]interface {} | len:2>{ "f:annotations": <map[string]interface {} | len:6>{ ".": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/instance.generation": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/instance.name": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/instance.uid": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/type": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/version": <map[string]interface {} | len:0>{}, }, "f:labels": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/part-of": <map[string]interface {} | len:0>{}, }, }, "f:spec": <map[string]interface {} | len:7>{ ".": <map[string]interface {} | len:0>{}, "f:flavorFungibility": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:whenCanBorrow": <map[string]interface {} | len:0>{}, "f:whenCanPreempt": <map[string]interface {} | len:0>{}, }, "f:namespaceSelector": <map[string]interface {} | len:0>{}, "f:preemption": <map[string]interface {} | len:4>{ ".": <map[string]interface {} | len:0>{}, "f:borrowWithinCohort": <map[string]interface {} | len:2>{...: ..., ...: ...}, "f:reclaimWithinCohort": <map[string]interface {} | len:0>{}, "f:withinClusterQueue": <map[string]interface {} | len:0>{}, }, "f:queueingStrategy": <map[string]interface {} | len:0>{}, "f:resourceGroups": <map[string]interface {} | len:0>{}, "f:stopPolicy": <map[string]interface {} | len:0>{}, }, }, }, ], "name": <string>"default", "resourceVersion": <string>"136026", "uid": <string>"b714554b-0d85-4628-a6f1-15547b6aee0d", "annotations": <map[string]interface {} | len:5>{ "platform.opendatahub.io/instance.generation": <string>"1", "platform.opendatahub.io/instance.name": <string>"default-kueue", "platform.opendatahub.io/instance.uid": <string>"952d240c-77ad-4fb9-b197-ebb6e0289954", "platform.opendatahub.io/type": <string>"OpenShift AI Self-Managed", "platform.opendatahub.io/version": <string>"3.5.0-ea.2", }, "creationTimestamp": <string>"2026-07-05T17:27:36Z", }, "spec": <map[string]interface {} | len:6>{ "flavorFungibility": <map[string]interface {} | len:2>{ "whenCanBorrow": <string>"MayStopSearch", "whenCanPreempt": <string>"TryNextFlavor", }, "namespaceSelector": <map[string]interface {} | len:1>{ "matchLabels": <map[string]interface {} | len:1>{ "kueue.openshift.io/managed": <string>"true", }, }, "preemption": <map[string]interface {} | len:3>{ "withinClusterQueue": <string>"Never", "borrowWithinCohort": <map[string]interface {} | len:1>{ "policy": <string>"Never", }, "reclaimWithinCohort": <string>"Never", }, "queueingStrategy": <string>"BestEffortFIFO", "resourceGroups": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:2>{ "coveredResources": <[]interface {} | len:2, cap:2>[<string>"cpu", <string>"memory"], "flavors": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:2>{ "name": <string>"default-flavor", "resources": <[]interface {} | len:2, cap:2>[ <map[string]interface {} | len:2>{...: ..., ...: ...}, <map[string]interface {} | len:2>{...: ..., ...: ...}, ], }, ], }, ], "stopPolicy": <string>"None", }, "apiVersion": <string>"kueue.x-k8s.io/v1beta2", "kind": <string>"ClusterQueue", }, }, ] to be empty test_context_test.go:1106: cleanup: best-effort delete failed for /, Kind= redhat-ods-applications/kueue-manager-config: Object 'Kind' is missing in 'unstructured object has no kind' --- FAIL: TestOdhOperator/components/group_2/kueue (380.95s) === RUN TestOdhOperator/components/group_2 --- FAIL: TestOdhOperator/components/group_2 (0.00s) === RUN TestOdhOperator/components/group_3/mlflowoperator/Validate_component_enabled 2026-07-05T17:32:45Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.21.23"}} testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_3/mlflowoperator/Validate_component_enabled (300.01s) === RUN TestOdhOperator/components/group_3/mlflowoperator === PAUSE TestOdhOperator/components/group_3/mlflowoperator === CONT TestOdhOperator/components/group_3/mlflowoperator resource_applier_test.go:209: Timed out after 300.009s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:285 with: Error occurred while applying the resource 'default-dsc' of kind 'DataScienceCluster': Internal error occurred: failed calling webhook "datasciencecluster-v2-defaulter.opendatahub.io": failed to call webhook: Post "https://rhods-operator-service.redhat-ods-operator.svc:443/mutate-datasciencecluster-v2?timeout=10s": no endpoints available for service "rhods-operator-service" Unexpected error: <*errors.StatusError | 0x29086d7850e0>: Internal error occurred: failed calling webhook "datasciencecluster-v2-defaulter.opendatahub.io": failed to call webhook: Post "https://rhods-operator-service.redhat-ods-operator.svc:443/mutate-datasciencecluster-v2?timeout=10s": no endpoints available for service "rhods-operator-service" { ErrStatus: { TypeMeta: {Kind: "Status", APIVersion: "v1"}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, ShardInfo: nil, }, Status: "Failure", Message: "Internal error occurred: failed calling webhook \"datasciencecluster-v2-defaulter.opendatahub.io\": failed to call webhook: Post \"https://rhods-operator-service.redhat-ods-operator.svc:443/mutate-datasciencecluster-v2?timeout=10s\": no endpoints available for service \"rhods-operator-service\"", Reason: "InternalError", Details: { Name: "", Group: "", Kind: "", UID: "", Causes: [ { Type: "", Message: "failed calling webhook \"datasciencecluster-v2-defaulter.opendatahub.io\": failed to call webhook: Post \"https://rhods-operator-service.redhat-ods-operator.svc:443/mutate-datasciencecluster-v2?timeout=10s\": no endpoints available for service \"rhods-operator-service\"", Field: "", }, ], RetryAfterSeconds: 0, }, Code: 500, }, } occurred --- FAIL: TestOdhOperator/components/group_3/mlflowoperator (300.06s) === RUN TestOdhOperator/components/group_3/trustyai/Validate_component_enabled testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_3/trustyai/Validate_component_enabled (600.01s) === RUN TestOdhOperator/components/group_3/trustyai === PAUSE TestOdhOperator/components/group_3/trustyai === CONT TestOdhOperator/components/group_3/trustyai 2026-07-05T17:32:45Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.21.23"}} resource_applier_test.go:209: Timed out after 600.008s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:285 with: Error occurred while applying the resource 'default-dsc' of kind 'DataScienceCluster': Internal error occurred: failed calling webhook "datasciencecluster-v2-defaulter.opendatahub.io": failed to call webhook: Post "https://rhods-operator-service.redhat-ods-operator.svc:443/mutate-datasciencecluster-v2?timeout=10s": no endpoints available for service "rhods-operator-service" Unexpected error: <*errors.StatusError | 0x29086d1dc8c0>: Internal error occurred: failed calling webhook "datasciencecluster-v2-defaulter.opendatahub.io": failed to call webhook: Post "https://rhods-operator-service.redhat-ods-operator.svc:443/mutate-datasciencecluster-v2?timeout=10s": no endpoints available for service "rhods-operator-service" { ErrStatus: { TypeMeta: {Kind: "Status", APIVersion: "v1"}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, ShardInfo: nil, }, Status: "Failure", Message: "Internal error occurred: failed calling webhook \"datasciencecluster-v2-defaulter.opendatahub.io\": failed to call webhook: Post \"https://rhods-operator-service.redhat-ods-operator.svc:443/mutate-datasciencecluster-v2?timeout=10s\": no endpoints available for service \"rhods-operator-service\"", Reason: "InternalError", Details: { Name: "", Group: "", Kind: "", UID: "", Causes: [ { Type: "", Message: "failed calling webhook \"datasciencecluster-v2-defaulter.opendatahub.io\": failed to call webhook: Post \"https://rhods-operator-service.redhat-ods-operator.svc:443/mutate-datasciencecluster-v2?timeout=10s\": no endpoints available for service \"rhods-operator-service\"", Field: "", }, ], RetryAfterSeconds: 0, }, Code: 500, }, } occurred --- FAIL: TestOdhOperator/components/group_3/trustyai (600.06s) === RUN TestOdhOperator/components/group_3 --- FAIL: TestOdhOperator/components/group_3 (0.00s) === RUN TestOdhOperator/components 2026/07/05 17:32:45 TEST FAILURE DETECTED: group 2 2026/07/05 17:32:45 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 2) === 2026/07/05 17:32:45 clusterhealth: node metrics unavailable (Metrics Server may not be installed): the server is currently unable to handle the request (get nodes.metrics.k8s.io) 2026/07/05 17:32:45 === CLUSTER STATE === 2026/07/05 17:32:45 Failed to collect node data: unhealthy nodes: ip-10-0-131-27.ec2.internal (Ready=Unknown); ip-10-0-133-110.ec2.internal (Ready=Unknown) 2026/07/05 17:32:45 === DEPLOYMENTS === 2026/07/05 17:32:45 Failed to collect deployment data: deployments not ready: redhat-ods-applications/dashboard-redirect (0/2), redhat-ods-applications/mlflow-operator-controller-manager (0/1), redhat-ods-applications/postgres (0/1), redhat-ods-applications/postgres-deployment (0/1), redhat-ods-applications/rhods-dashboard (0/2), redhat-ods-applications/trustyai-service-operator-controller-manager (0/1) 2026/07/05 17:32:45 === PODS === 2026/07/05 17:32:45 Failed to collect pod data: redhat-ods-applications/mlflow-mg-3130-g1-qwbnh: container combine-ca-bundles Completed (exit 0) 2026/07/05 17:32:45 === OPERATOR STATUS === 2026/07/05 17:32:45 Failed to collect operator data: operator deployment rhods-operator: 0/3 ready; dependent kueue: deployment 0/2 ready; dependent jobset: deployment 0/1 ready; dependent leader-worker-set: deployment 0/1 ready; dependent cert-manager: deployment 0/1 ready; dependent kuadrant: deployment 0/1 ready 2026/07/05 17:32:45 === DSCI STATUS === 2026/07/05 17:32:45 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/05 17:32:45 === DSC STATUS === 2026/07/05 17:32:45 Failed to collect DSC data: default-dsc unhealthy conditions: ModelsAsServiceReady=False 2026/07/05 17:32:45 === RECENT EVENTS === 2026/07/05 17:32:45 17:32:11 WARN Pod/dashboard-redirect-7bf9c448b9-4cjxz: NodeNotReady - Node is not ready 2026/07/05 17:32:45 17:32:11 WARN Pod/kube-apiserver-proxy-ip-10-0-133-110.ec2.internal: NodeNotReady - Node is not ready 2026/07/05 17:32:45 17:32:11 INFO Pod/mlflow-mg-3130-g1-qwbnh: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/mlflow-mg-3130-g1-qwbnh 2026/07/05 17:32:45 17:32:11 INFO Pod/postgres-69fb4594cf-k2pzk: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/postgres-69fb4594cf-k2pzk 2026/07/05 17:32:45 17:32:11 INFO Pod/postgres-deployment-86754cb759-hdpjr: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/postgres-deployment-86754cb759-hdpjr 2026/07/05 17:32:45 17:32:11 WARN Pod/konnectivity-agent-v58s8: NodeNotReady - Node is not ready 2026/07/05 17:32:45 17:32:11 WARN Pod/rhods-dashboard-5fd8c7f95b-hfhp8: NodeNotReady - Node is not ready 2026/07/05 17:32:45 17:32:11 INFO Pod/trustyai-service-operator-controller-manager-7bcdbcb4fd-vlmrl: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/trustyai-service-operator-controller-manager-7bcdbcb4fd-vlmrl 2026/07/05 17:32:45 17:32:10 WARN Pod/dashboard-redirect-7bf9c448b9-662mp: NodeNotReady - Node is not ready 2026/07/05 17:32:45 17:32:10 WARN Pod/global-pull-secret-syncer-fj4jm: NodeNotReady - Node is not ready 2026/07/05 17:32:45 17:32:10 WARN Pod/global-pull-secret-syncer-kg7gm: NodeNotReady - Node is not ready 2026/07/05 17:32:45 17:32:10 WARN Pod/kube-apiserver-proxy-ip-10-0-131-27.ec2.internal: NodeNotReady - Node is not ready 2026/07/05 17:32:45 17:32:10 WARN Pod/mlflow-operator-controller-manager-75654dffb8-4pndj: NodeNotReady - Node is not ready 2026/07/05 17:32:45 17:32:10 WARN Pod/rhods-dashboard-5fd8c7f95b-6p9g2: NodeNotReady - Node is not ready 2026/07/05 17:32:45 17:32:10 WARN Pod/konnectivity-agent-vfm27: NodeNotReady - Node is not ready 2026/07/05 17:32:45 17:32:00 WARN Pod/postgres-69fb4594cf-k2pzk: NodeNotReady - Node is not ready 2026/07/05 17:32:45 17:31:59 WARN Pod/global-pull-secret-syncer-f2zfw: NodeNotReady - Node is not ready 2026/07/05 17:32:45 17:31:59 WARN Pod/postgres-deployment-86754cb759-hdpjr: NodeNotReady - Node is not ready 2026/07/05 17:32:45 17:31:59 WARN Pod/kube-apiserver-proxy-ip-10-0-130-40.ec2.internal: NodeNotReady - Node is not ready 2026/07/05 17:32:45 17:31:59 WARN Pod/mlflow-mg-3130-g1-qwbnh: NodeNotReady - Node is not ready 2026/07/05 17:32:45 17:31:59 WARN Pod/konnectivity-agent-sqqtj: NodeNotReady - Node is not ready 2026/07/05 17:32:45 17:31:59 WARN Pod/trustyai-service-operator-controller-manager-7bcdbcb4fd-vlmrl: NodeNotReady - Node is not ready 2026/07/05 17:32:45 17:31:14 WARN Pod/mlflow-operator-controller-manager-75654dffb8-4pndj: Unhealthy - Readiness probe failed: Get "http://10.132.0.40:8081/readyz": dial tcp 10.132.0.40:8081: connect: connection refused 2026/07/05 17:32:45 17:31:13 INFO Pod/mlflow-mg-3130-g1-qwbnh: Started - Started container db-migrate 2026/07/05 17:32:45 17:31:13 INFO Pod/mlflow-mg-3130-g1-qwbnh: Created - Created container: db-migrate 2026/07/05 17:32:45 17:31:13 INFO Pod/mlflow-mg-3130-g1-qwbnh: Pulled - Successfully pulled image "quay.io/rhoai/odh-mlflow-rhel9@sha256:8f6927d958c24ce56e70977707862824e933e75ced2609eed5bbcd36847a85ce" in 303ms (303ms including waiting). Image size: 1218048430 bytes. 2026/07/05 17:32:45 17:31:12 INFO Pod/mlflow-mg-3130-g1-qwbnh: Pulling - Pulling image "quay.io/rhoai/odh-mlflow-rhel9@sha256:8f6927d958c24ce56e70977707862824e933e75ced2609eed5bbcd36847a85ce" 2026/07/05 17:32:45 17:31:11 INFO Pod/mlflow-mg-3130-g1-qwbnh: Started - Started container combine-ca-bundles 2026/07/05 17:32:45 17:31:11 INFO Pod/mlflow-mg-3130-g1-qwbnh: Created - Created container: combine-ca-bundles 2026/07/05 17:32:45 17:31:11 INFO Pod/mlflow-mg-3130-g1-qwbnh: Pulled - Successfully pulled image "quay.io/rhoai/odh-mlflow-rhel9@sha256:8f6927d958c24ce56e70977707862824e933e75ced2609eed5bbcd36847a85ce" in 13.58s (13.58s including waiting). Image size: 1218048430 bytes. 2026/07/05 17:32:45 17:30:58 INFO Pod/mlflow-mg-3130-g1-qwbnh: AddedInterface - Add eth0 [10.133.0.105/23] from ovn-kubernetes 2026/07/05 17:32:45 17:30:58 INFO Pod/mlflow-mg-3130-g1-qwbnh: Pulling - Pulling image "quay.io/rhoai/odh-mlflow-rhel9@sha256:8f6927d958c24ce56e70977707862824e933e75ced2609eed5bbcd36847a85ce" 2026/07/05 17:32:45 17:30:53 INFO Pod/mlflow-mg-3130-g1-qwbnh: SuccessfulAttachVolume - AttachVolume.Attach succeeded for volume "pvc-0c1628ed-380f-4ecd-8904-6db259316b5a" 2026/07/05 17:32:45 17:30:51 INFO Pod/mlflow-mg-3130-g1-qwbnh: Scheduled - Successfully assigned redhat-ods-applications/mlflow-mg-3130-g1-qwbnh to ip-10-0-130-40.ec2.internal 2026/07/05 17:32:45 17:30:50 INFO PersistentVolumeClaim/mlflow-pvc: ProvisioningSucceeded - Successfully provisioned volume pvc-0c1628ed-380f-4ecd-8904-6db259316b5a 2026/07/05 17:32:45 17:30:48 INFO PersistentVolumeClaim/mlflow-pvc: Provisioning - External provisioner is provisioning volume for claim "redhat-ods-applications/mlflow-pvc" 2026/07/05 17:32:45 17:30:48 INFO PersistentVolumeClaim/mlflow-pvc: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. 2026/07/05 17:32:45 17:30:48 INFO PersistentVolumeClaim/mlflow-pvc: WaitForFirstConsumer - waiting for first consumer to be created before binding 2026/07/05 17:32:45 17:30:48 INFO Job/mlflow-mg-3130-g1: SuccessfulCreate - Created pod: mlflow-mg-3130-g1-qwbnh 2026/07/05 17:32:45 17:30:44 INFO Pod/postgres-deployment-86754cb759-hdpjr: Pulled - Successfully pulled image "registry.redhat.io/rhel9/postgresql-15@sha256:90ec347a35ab8a5d530c8d09f5347b13cc71df04f3b994bfa8b1a409b1171d59" in 5.209s (5.209s including waiting). Image size: 404709194 bytes. 2026/07/05 17:32:45 17:30:44 INFO Pod/postgres-deployment-86754cb759-hdpjr: Created - Created container: postgres 2026/07/05 17:32:45 17:30:44 INFO Pod/postgres-deployment-86754cb759-hdpjr: Started - Started container postgres 2026/07/05 17:32:45 17:30:39 INFO Pod/postgres-deployment-86754cb759-hdpjr: AddedInterface - Add eth0 [10.133.0.104/23] from ovn-kubernetes 2026/07/05 17:32:45 17:30:39 INFO Pod/postgres-deployment-86754cb759-hdpjr: Pulling - Pulling image "registry.redhat.io/rhel9/postgresql-15@sha256:90ec347a35ab8a5d530c8d09f5347b13cc71df04f3b994bfa8b1a409b1171d59" 2026/07/05 17:32:45 17:30:38 INFO Pod/postgres-deployment-86754cb759-hdpjr: SuccessfulAttachVolume - AttachVolume.Attach succeeded for volume "pvc-e35d88c4-067c-4878-81a1-d533ce429255" 2026/07/05 17:32:45 17:30:36 INFO Pod/postgres-deployment-86754cb759-hdpjr: Scheduled - Successfully assigned redhat-ods-applications/postgres-deployment-86754cb759-hdpjr to ip-10-0-130-40.ec2.internal 2026/07/05 17:32:45 17:30:35 INFO PersistentVolumeClaim/postgres-pvc: ProvisioningSucceeded - Successfully provisioned volume pvc-e35d88c4-067c-4878-81a1-d533ce429255 2026/07/05 17:32:45 17:30:33 INFO PersistentVolumeClaim/postgres-pvc: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. 2026/07/05 17:32:45 17:30:33 INFO PersistentVolumeClaim/postgres-pvc: WaitForFirstConsumer - waiting for first consumer to be created before binding 2026/07/05 17:32:45 17:30:33 INFO ReplicaSet/postgres-deployment-86754cb759: SuccessfulCreate - Created pod: postgres-deployment-86754cb759-hdpjr 2026/07/05 17:32:45 17:30:33 INFO Deployment/postgres-deployment: ScalingReplicaSet - Scaled up replica set postgres-deployment-86754cb759 from 0 to 1 2026/07/05 17:32:45 17:30:33 INFO PersistentVolumeClaim/postgres-pvc: Provisioning - External provisioner is provisioning volume for claim "redhat-ods-applications/postgres-pvc" 2026/07/05 17:32:45 17:30:08 INFO Pod/maas-api-557679ff4c-q962c: Killing - Stopping container maas-api 2026/07/05 17:32:45 17:30:07 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:32:45 17:30:07 INFO Pod/maas-controller-5b858f66fc-kqp97: Killing - Stopping container manager 2026/07/05 17:32:45 17:30:07 INFO Pod/model-registry-operator-controller-manager-7b4b7bd8cc-dq7lv: Killing - Stopping container manager 2026/07/05 17:32:45 17:30:07 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-4xg4w: Killing - Stopping container manager 2026/07/05 17:32:45 17:30:07 INFO Pod/kserve-controller-manager-588b4b787c-hznbb: Killing - Stopping container manager 2026/07/05 17:32:45 17:30:07 INFO Pod/llmisvc-controller-manager-bc44d7c7b-ngq98: Killing - Stopping container manager 2026/07/05 17:32:45 17:30:07 INFO Pod/model-serving-api-5c897df776-brxf9: Killing - Stopping container server 2026/07/05 17:32:45 17:30:07 INFO Pod/odh-model-controller-566d8764f4-hqcjr: Killing - Stopping container manager 2026/07/05 17:32:45 17:30:06 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:32:45 17:30:06 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:32:45 17:30:06 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:32:45 17:30:05 INFO Job/maas-api-key-cleanup-29721210: Completed - Job completed 2026/07/05 17:32:45 17:30:05 INFO CronJob/maas-api-key-cleanup: SawCompletedJob - Saw completed job: maas-api-key-cleanup-29721210, condition: Complete 2026/07/05 17:32:45 17:30:00 INFO Pod/maas-api-key-cleanup-29721210-z9rfr: Scheduled - Successfully assigned redhat-ods-applications/maas-api-key-cleanup-29721210-z9rfr to ip-10-0-130-40.ec2.internal 2026/07/05 17:32:45 17:30:00 INFO CronJob/maas-api-key-cleanup: SuccessfulCreate - Created job maas-api-key-cleanup-29721210 2026/07/05 17:32:45 17:30:00 INFO Job/maas-api-key-cleanup-29721210: SuccessfulCreate - Created pod: maas-api-key-cleanup-29721210-z9rfr 2026/07/05 17:32:45 17:30:00 INFO Pod/maas-api-key-cleanup-29721210-z9rfr: Started - Started container cleanup 2026/07/05 17:32:45 17:30:00 INFO Pod/maas-api-key-cleanup-29721210-z9rfr: Created - Created container: cleanup 2026/07/05 17:32:45 17:30:00 INFO Pod/maas-api-key-cleanup-29721210-z9rfr: Pulled - Container image "registry.redhat.io/ubi9/ubi-minimal@sha256:907b68736aa798b2d38255b7aa070b2a70acb90803864a40f05d0ec47556ddd0" already present on machine 2026/07/05 17:32:45 17:30:00 INFO Pod/maas-api-key-cleanup-29721210-z9rfr: AddedInterface - Add eth0 [10.133.0.103/23] from ovn-kubernetes 2026/07/05 17:32:45 17:29:53 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-67777f585d-4xg4w_dcdf6243-cd1d-404a-8e8e-9ff9b41ca05d became leader 2026/07/05 17:32:45 17:29:53 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-566d8764f4-hqcjr_a2fc95de-5205-4cff-9ff4-ef7d8358cba4 became leader 2026/07/05 17:32:45 17:29:52 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-4xg4w: Started - Started container manager 2026/07/05 17:32:45 17:29:52 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-4xg4w: Created - Created container: manager 2026/07/05 17:32:45 17:29:52 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-4xg4w: Pulled - Successfully pulled image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:79fa44b245109a0c1d214c8b313dc8172b367dedf75c4691ed1faefa46f41775" in 1.887s (1.887s including waiting). Image size: 195240092 bytes. 2026/07/05 17:32:45 17:29:51 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-4xg4w: Pulling - Pulling image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:79fa44b245109a0c1d214c8b313dc8172b367dedf75c4691ed1faefa46f41775" 2026/07/05 17:32:45 17:29:50 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-4xg4w: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-67777f585d-4xg4w to ip-10-0-130-40.ec2.internal 2026/07/05 17:32:45 17:29:50 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-67777f585d from 0 to 1 2026/07/05 17:32:45 17:29:50 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-67777f585d: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-67777f585d-4xg4w 2026/07/05 17:32:45 17:29:50 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-9n5tj: Killing - Stopping container manager 2026/07/05 17:32:45 17:29:50 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-5zdfm: Killing - Stopping container manager 2026/07/05 17:32:45 17:29:50 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-4xg4w: AddedInterface - Add eth0 [10.133.0.102/23] from ovn-kubernetes 2026/07/05 17:32:45 17:29:48 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:32:45 17:29:48 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:32:45 17:29:48 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: Created - Created container: mlflow-ui 2026/07/05 17:32:45 17:29:48 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:223cf7fb6040bbf57b2445d0bc6a1b5043d4bcf54c3bd272700791594a6c2623" in 435ms (435ms including waiting). Image size: 185088364 bytes. 2026/07/05 17:32:45 17:29:48 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:223cf7fb6040bbf57b2445d0bc6a1b5043d4bcf54c3bd272700791594a6c2623" 2026/07/05 17:32:45 17:29:48 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: Started - Started container mlflow-ui 2026/07/05 17:32:45 17:29:48 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:223cf7fb6040bbf57b2445d0bc6a1b5043d4bcf54c3bd272700791594a6c2623" in 464ms (464ms including waiting). Image size: 185088364 bytes. 2026/07/05 17:32:45 17:29:48 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:223cf7fb6040bbf57b2445d0bc6a1b5043d4bcf54c3bd272700791594a6c2623" 2026/07/05 17:32:45 17:29:48 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:feee294f8bab936cc5684e267a4a065c01ebb5d554ca88cfe870fc61b9e0a01d" in 477ms (477ms including waiting). Image size: 177813676 bytes. 2026/07/05 17:32:45 17:29:48 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: Created - Created container: mlflow-ui 2026/07/05 17:32:45 17:29:48 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: Started - Started container mlflow-ui 2026/07/05 17:32:45 17:29:48 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:feee294f8bab936cc5684e267a4a065c01ebb5d554ca88cfe870fc61b9e0a01d" in 537ms (537ms including waiting). Image size: 177813676 bytes. 2026/07/05 17:32:45 17:29:47 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: Started - Started container maas-ui 2026/07/05 17:32:45 17:29:47 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:feee294f8bab936cc5684e267a4a065c01ebb5d554ca88cfe870fc61b9e0a01d" 2026/07/05 17:32:45 17:29:47 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: Created - Created container: gen-ai-ui 2026/07/05 17:32:45 17:29:47 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:b3572eaf1dcda4027027901e592df26307a6135b9c5b4778d15a711bef52c280" in 344ms (344ms including waiting). Image size: 241238855 bytes. 2026/07/05 17:32:45 17:29:47 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: Started - Started container gen-ai-ui 2026/07/05 17:32:45 17:29:47 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:7052d754cf9ab54cf8e9b09bfbc58b888aa0e5c3f60bbd49abd46a97f2fad9b3" in 431ms (431ms including waiting). Image size: 187798114 bytes. 2026/07/05 17:32:45 17:29:47 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: Created - Created container: maas-ui 2026/07/05 17:32:45 17:29:47 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:b3572eaf1dcda4027027901e592df26307a6135b9c5b4778d15a711bef52c280" in 450ms (450ms including waiting). Image size: 241238855 bytes. 2026/07/05 17:32:45 17:29:47 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:7052d754cf9ab54cf8e9b09bfbc58b888aa0e5c3f60bbd49abd46a97f2fad9b3" 2026/07/05 17:32:45 17:29:47 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:feee294f8bab936cc5684e267a4a065c01ebb5d554ca88cfe870fc61b9e0a01d" 2026/07/05 17:32:45 17:29:47 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: Started - Started container maas-ui 2026/07/05 17:32:45 17:29:47 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: Created - Created container: maas-ui 2026/07/05 17:32:45 17:29:47 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:7052d754cf9ab54cf8e9b09bfbc58b888aa0e5c3f60bbd49abd46a97f2fad9b3" in 496ms (496ms including waiting). Image size: 187798114 bytes. 2026/07/05 17:32:45 17:29:47 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:7052d754cf9ab54cf8e9b09bfbc58b888aa0e5c3f60bbd49abd46a97f2fad9b3" 2026/07/05 17:32:45 17:29:47 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: Started - Started container gen-ai-ui 2026/07/05 17:32:45 17:29:47 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: Created - Created container: gen-ai-ui 2026/07/05 17:32:45 17:29:46 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: Created - Created container: model-registry-ui 2026/07/05 17:32:45 17:29:46 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: Created - Created container: model-registry-ui 2026/07/05 17:32:45 17:29:46 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: Started - Started container model-registry-ui 2026/07/05 17:32:45 17:29:46 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:b3572eaf1dcda4027027901e592df26307a6135b9c5b4778d15a711bef52c280" 2026/07/05 17:32:45 17:29:46 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:a87537db8e31c592fff6c05e0a823e0c37f6f6426d19d69207e15f515476c928" in 627ms (627ms including waiting). Image size: 182434600 bytes. 2026/07/05 17:32:45 17:29:46 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:a87537db8e31c592fff6c05e0a823e0c37f6f6426d19d69207e15f515476c928" 2026/07/05 17:32:45 17:29:46 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: Started - Started container kube-rbac-proxy 2026/07/05 17:32:45 17:29:46 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: Created - Created container: kube-rbac-proxy 2026/07/05 17:32:45 17:29:46 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: Started - Started container rhods-dashboard 2026/07/05 17:32:45 17:29:46 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: Pulled - Container image "quay.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:ac5537f459cd812a12e8f6ae13001150e29954913007b1790a44b15fb75f4926" already present on machine 2026/07/05 17:32:45 17:29:46 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: Created - Created container: kube-rbac-proxy 2026/07/05 17:32:45 17:29:46 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: Started - Started container kube-rbac-proxy 2026/07/05 17:32:45 17:29:46 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:a87537db8e31c592fff6c05e0a823e0c37f6f6426d19d69207e15f515476c928" 2026/07/05 17:32:45 17:29:46 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:a87537db8e31c592fff6c05e0a823e0c37f6f6426d19d69207e15f515476c928" in 306ms (306ms including waiting). Image size: 182434600 bytes. 2026/07/05 17:32:45 17:29:46 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: Pulled - Successfully pulled image "quay.io/rhoai/odh-dashboard-rhel9@sha256:29d00e57fc50806281cb490815e8dda6a58845704d5d8ff4ec5444fee55e0a67" in 427ms (427ms including waiting). Image size: 799818131 bytes. 2026/07/05 17:32:45 17:29:46 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: Started - Started container model-registry-ui 2026/07/05 17:32:45 17:29:46 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:b3572eaf1dcda4027027901e592df26307a6135b9c5b4778d15a711bef52c280" 2026/07/05 17:32:45 17:29:46 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: Created - Created container: rhods-dashboard 2026/07/05 17:32:45 17:29:45 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-5fd8c7f95b-hfhp8 to ip-10-0-131-27.ec2.internal 2026/07/05 17:32:45 17:29:45 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: Pulled - Container image "quay.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:ac5537f459cd812a12e8f6ae13001150e29954913007b1790a44b15fb75f4926" already present on machine 2026/07/05 17:32:45 17:29:45 INFO ReplicaSet/rhods-dashboard-5fd8c7f95b: SuccessfulCreate - Created pod: rhods-dashboard-5fd8c7f95b-hfhp8 2026/07/05 17:32:45 17:29:45 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: AddedInterface - Add eth0 [10.132.0.89/23] from ovn-kubernetes 2026/07/05 17:32:45 17:29:45 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: Pulled - Successfully pulled image "quay.io/rhoai/odh-dashboard-rhel9@sha256:29d00e57fc50806281cb490815e8dda6a58845704d5d8ff4ec5444fee55e0a67" in 540ms (540ms including waiting). Image size: 799818131 bytes. 2026/07/05 17:32:45 17:29:45 INFO Pod/rhods-dashboard-5fd8c7f95b-hfhp8: Pulling - Pulling image "quay.io/rhoai/odh-dashboard-rhel9@sha256:29d00e57fc50806281cb490815e8dda6a58845704d5d8ff4ec5444fee55e0a67" 2026/07/05 17:32:45 17:29:45 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: Created - Created container: rhods-dashboard 2026/07/05 17:32:45 17:29:45 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: Pulling - Pulling image "quay.io/rhoai/odh-dashboard-rhel9@sha256:29d00e57fc50806281cb490815e8dda6a58845704d5d8ff4ec5444fee55e0a67" 2026/07/05 17:32:45 17:29:45 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: AddedInterface - Add eth0 [10.134.0.83/23] from ovn-kubernetes 2026/07/05 17:32:45 17:29:45 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: Started - Started container rhods-dashboard 2026/07/05 17:32:45 17:29:44 INFO Pod/rhods-dashboard-5fd8c7f95b-6p9g2: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-5fd8c7f95b-6p9g2 to ip-10-0-133-110.ec2.internal 2026/07/05 17:32:45 17:29:44 INFO ReplicaSet/rhods-dashboard-5fd8c7f95b: SuccessfulCreate - Created pod: rhods-dashboard-5fd8c7f95b-6p9g2 2026/07/05 17:32:45 17:29:44 INFO Deployment/rhods-dashboard: ScalingReplicaSet - Scaled up replica set rhods-dashboard-5fd8c7f95b from 0 to 2 2026/07/05 17:32:45 17:29:44 WARN ReplicaSet/rhods-dashboard-5fd8c7f95b: FailedCreate - Error creating: pods "rhods-dashboard-5fd8c7f95b-" is forbidden: error looking up service account redhat-ods-applications/rhods-dashboard: serviceaccount "rhods-dashboard" not found 2026/07/05 17:32:45 17:29:42 INFO Pod/dashboard-redirect-7bf9c448b9-4cjxz: Scheduled - Successfully assigned redhat-ods-applications/dashboard-redirect-7bf9c448b9-4cjxz to ip-10-0-131-27.ec2.internal 2026/07/05 17:32:45 17:29:42 INFO Pod/dashboard-redirect-7bf9c448b9-662mp: Scheduled - Successfully assigned redhat-ods-applications/dashboard-redirect-7bf9c448b9-662mp to ip-10-0-133-110.ec2.internal 2026/07/05 17:32:45 17:29:42 INFO ReplicaSet/dashboard-redirect-7bf9c448b9: SuccessfulCreate - Created pod: dashboard-redirect-7bf9c448b9-4cjxz 2026/07/05 17:32:45 17:29:42 INFO Pod/dashboard-redirect-7bf9c448b9-4cjxz: Created - Created container: nginx 2026/07/05 17:32:45 17:29:42 INFO Pod/dashboard-redirect-7bf9c448b9-662mp: Started - Started container nginx 2026/07/05 17:32:45 17:29:42 INFO Pod/dashboard-redirect-7bf9c448b9-662mp: AddedInterface - Add eth0 [10.134.0.82/23] from ovn-kubernetes 2026/07/05 17:32:45 17:29:42 INFO Pod/dashboard-redirect-7bf9c448b9-4cjxz: AddedInterface - Add eth0 [10.132.0.88/23] from ovn-kubernetes 2026/07/05 17:32:45 17:29:42 INFO Pod/dashboard-redirect-7bf9c448b9-662mp: Pulled - Container image "registry.redhat.io/ubi9/nginx-126@sha256:f0a79ccf21b8780a7534d78ee0e49d1852654f5f0f52e17ebdbe185a71d93253" already present on machine 2026/07/05 17:32:45 17:29:42 INFO Pod/dashboard-redirect-7bf9c448b9-4cjxz: Pulled - Container image "registry.redhat.io/ubi9/nginx-126@sha256:f0a79ccf21b8780a7534d78ee0e49d1852654f5f0f52e17ebdbe185a71d93253" already present on machine 2026/07/05 17:32:45 17:29:42 INFO Pod/dashboard-redirect-7bf9c448b9-662mp: Created - Created container: nginx 2026/07/05 17:32:45 17:29:42 INFO Pod/dashboard-redirect-7bf9c448b9-4cjxz: Started - Started container nginx 2026/07/05 17:32:45 17:29:41 INFO Deployment/dashboard-redirect: ScalingReplicaSet - Scaled up replica set dashboard-redirect-7bf9c448b9 from 0 to 2 2026/07/05 17:32:45 17:29:41 INFO ReplicaSet/dashboard-redirect-7bf9c448b9: SuccessfulCreate - Created pod: dashboard-redirect-7bf9c448b9-662mp 2026/07/05 17:32:45 17:29:36 INFO Pod/odh-model-controller-566d8764f4-hqcjr: Started - Started container manager 2026/07/05 17:32:45 17:29:36 INFO Pod/odh-model-controller-566d8764f4-hqcjr: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:49111a3777043a2d16e6599bdc99cacb4839a64949097bf9bdfa798d3bba2c43" in 435ms (435ms including waiting). Image size: 238959021 bytes. 2026/07/05 17:32:45 17:29:36 INFO Pod/odh-model-controller-566d8764f4-hqcjr: Created - Created container: manager 2026/07/05 17:32:45 17:29:35 INFO Pod/odh-model-controller-566d8764f4-hqcjr: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-566d8764f4-hqcjr to ip-10-0-131-27.ec2.internal 2026/07/05 17:32:45 17:29:35 INFO Pod/odh-model-controller-566d8764f4-nmggg: Killing - Stopping container manager 2026/07/05 17:32:45 17:29:35 INFO ReplicaSet/odh-model-controller-566d8764f4: SuccessfulCreate - Created pod: odh-model-controller-566d8764f4-hqcjr 2026/07/05 17:32:45 17:29:35 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-566d8764f4 from 0 to 1 2026/07/05 17:32:45 17:29:35 INFO Pod/odh-model-controller-566d8764f4-hqcjr: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:49111a3777043a2d16e6599bdc99cacb4839a64949097bf9bdfa798d3bba2c43" 2026/07/05 17:32:45 17:29:35 INFO Pod/odh-model-controller-566d8764f4-hqcjr: AddedInterface - Add eth0 [10.132.0.87/23] from ovn-kubernetes 2026/07/05 17:32:45 17:29:35 INFO Pod/odh-model-controller-566d8764f4-7fd6m: Killing - Stopping container manager 2026/07/05 17:32:45 17:29:21 INFO Pod/model-serving-api-5c897df776-brxf9: AddedInterface - Add eth0 [10.132.0.86/23] from ovn-kubernetes 2026/07/05 17:32:45 17:29:21 INFO Pod/model-serving-api-5c897df776-brxf9: Started - Started container server 2026/07/05 17:32:45 17:29:21 INFO Pod/model-serving-api-5c897df776-brxf9: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:c690422f9b5d3fc40fe327208b7af310035080730395bb9c4e393965dce7b53f" already present on machine 2026/07/05 17:32:45 17:29:21 INFO Pod/model-serving-api-5c897df776-brxf9: Created - Created container: server 2026/07/05 17:32:45 17:29:20 INFO Pod/model-serving-api-5c897df776-brxf9: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5c897df776-brxf9 to ip-10-0-131-27.ec2.internal 2026/07/05 17:32:45 17:29:20 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5c897df776 from 0 to 1 2026/07/05 17:32:45 17:29:20 INFO ReplicaSet/model-serving-api-5c897df776: SuccessfulCreate - Created pod: model-serving-api-5c897df776-brxf9 2026/07/05 17:32:45 17:29:20 INFO Pod/model-serving-api-5c897df776-ssdx7: Killing - Stopping container server 2026/07/05 17:32:45 17:29:20 INFO Pod/model-serving-api-5c897df776-vbs9m: Killing - Stopping container server 2026/07/05 17:32:45 17:28:22 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-9n5tj: Started - Started container manager 2026/07/05 17:32:45 17:28:22 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-9n5tj: Created - Created container: manager 2026/07/05 17:32:45 17:28:22 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-9n5tj: Pulled - Successfully pulled image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:79fa44b245109a0c1d214c8b313dc8172b367dedf75c4691ed1faefa46f41775" in 1.981s (1.981s including waiting). Image size: 195240092 bytes. 2026/07/05 17:32:45 17:28:20 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-9n5tj: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-67777f585d-9n5tj to ip-10-0-133-110.ec2.internal 2026/07/05 17:32:45 17:28:20 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-67777f585d: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-67777f585d-9n5tj 2026/07/05 17:32:45 17:28:20 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-9n5tj: AddedInterface - Add eth0 [10.134.0.81/23] from ovn-kubernetes 2026/07/05 17:32:45 17:28:20 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-67777f585d from 1 to 2 2026/07/05 17:32:45 17:28:20 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-9n5tj: Pulling - Pulling image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:79fa44b245109a0c1d214c8b313dc8172b367dedf75c4691ed1faefa46f41775" 2026/07/05 17:32:45 17:28:03 INFO Pod/odh-model-controller-566d8764f4-7fd6m: Created - Created container: manager 2026/07/05 17:32:45 17:28:03 INFO Pod/odh-model-controller-566d8764f4-7fd6m: Started - Started container manager 2026/07/05 17:32:45 17:28:02 INFO Pod/odh-model-controller-566d8764f4-7fd6m: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:49111a3777043a2d16e6599bdc99cacb4839a64949097bf9bdfa798d3bba2c43" in 2.404s (2.404s including waiting). Image size: 238959021 bytes. 2026/07/05 17:32:45 17:28:00 INFO Pod/odh-model-controller-566d8764f4-7fd6m: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-566d8764f4-7fd6m to ip-10-0-133-110.ec2.internal 2026/07/05 17:32:45 17:28:00 INFO Pod/odh-model-controller-566d8764f4-7fd6m: AddedInterface - Add eth0 [10.134.0.80/23] from ovn-kubernetes 2026/07/05 17:32:45 17:28:00 INFO Pod/odh-model-controller-566d8764f4-7fd6m: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:49111a3777043a2d16e6599bdc99cacb4839a64949097bf9bdfa798d3bba2c43" 2026/07/05 17:32:45 17:28:00 INFO ReplicaSet/odh-model-controller-566d8764f4: SuccessfulCreate - Created pod: odh-model-controller-566d8764f4-7fd6m 2026/07/05 17:32:45 17:28:00 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-566d8764f4 from 1 to 2 2026/07/05 17:32:45 === RESOURCE QUOTAS === 2026/07/05 17:32:45 No resource quotas found in monitored namespaces 2026/07/05 17:32:45 Classification for group 2: infrastructure/probe-failure (code=1014, confidence=medium) 2026/07/05 17:32:45 Evidence: event Pod/mlflow-operator-controller-manager-75654dffb8-4pndj: Unhealthy - Readiness probe failed: Get "http://10.132.0.40:8081/readyz": dial tcp 10.132.0.40:8081: connect: connection refused FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"probe-failure","error_code":1014,"evidence":["event Pod/mlflow-operator-controller-manager-75654dffb8-4pndj: Unhealthy - Readiness probe failed: Get \"http://10.132.0.40:8081/readyz\": dial tcp 10.132.0.40:8081: connect: connection refused"],"confidence":"medium"} 2026/07/05 17:32:45 Diagnostics complete for failed test: group 2 controller_test.go:763: Stopping: group 2 test failed. 2026/07/05 17:32:45 Circuit breaker: failure classified as infrastructure/probe-failure (code=1014) 2026/07/05 17:32:45 Circuit breaker: infrastructure/unknown failure recorded (2/3) 2026/07/05 17:42:45 TEST FAILURE DETECTED: group 3 2026/07/05 17:42:45 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 3) === 2026/07/05 17:42:46 === CLUSTER STATE === 2026/07/05 17:42:46 No nodes found in cluster 2026/07/05 17:42:46 === DEPLOYMENTS === 2026/07/05 17:42:46 Failed to collect deployment data: deployments not ready: redhat-ods-applications/dashboard-redirect (0/2), redhat-ods-applications/mlflow-operator-controller-manager (0/1), redhat-ods-applications/postgres (0/1), redhat-ods-applications/postgres-deployment (0/1), redhat-ods-applications/rhods-dashboard (0/2), redhat-ods-applications/trustyai-service-operator-controller-manager (0/1) 2026/07/05 17:42:46 === PODS === 2026/07/05 17:42:46 Failed to collect pod data: redhat-ods-applications/dashboard-redirect-7bf9c448b9-msvwd: phase=Pending; redhat-ods-applications/mlflow-operator-controller-manager-75654dffb8-wkr46: phase=Pending; redhat-ods-applications/postgres-69fb4594cf-pclwj: phase=Pending; redhat-ods-applications/rhods-dashboard-5fd8c7f95b-9svc7: phase=Pending; redhat-ods-applications/trustyai-service-operator-controller-manager-7bcdbcb4fd-xs5lx: phase=Pending 2026/07/05 17:42:46 === OPERATOR STATUS === 2026/07/05 17:42:46 Failed to collect operator data: operator deployment rhods-operator: 0/3 ready; operator pod rhods-operator-86ccbf48b5-2kszh: Pending; operator pod rhods-operator-86ccbf48b5-hcm7c: Pending; dependent kueue: deployment 0/2 ready; pod openshift-kueue-operator-8568d5f8-z45rd: Pending; dependent jobset: deployment 0/1 ready; pod jobset-operator-56d489b4b8-nbshb: Pending; dependent leader-worker-set: deployment 0/1 ready; dependent cert-manager: deployment 0/1 ready; dependent kuadrant: deployment 0/1 ready; pod authorino-operator-6d85f6564-tbbgz: Pending 2026/07/05 17:42:46 === DSCI STATUS === 2026/07/05 17:42:46 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/05 17:42:46 === DSC STATUS === 2026/07/05 17:42:46 Failed to collect DSC data: default-dsc unhealthy conditions: ModelsAsServiceReady=False 2026/07/05 17:42:46 === RECENT EVENTS === 2026/07/05 17:42:46 17:41:49 WARN Job/mlflow-mg-3130-g1: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/05 17:42:46 17:40:48 WARN ReplicaSet/dashboard-redirect-7bf9c448b9: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/05 17:42:46 17:40:48 WARN ReplicaSet/rhods-dashboard-5fd8c7f95b: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/05 17:42:46 17:40:48 WARN ReplicaSet/postgres-deployment-86754cb759: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/05 17:42:46 17:39:07 WARN Pod/mlflow-operator-controller-manager-75654dffb8-wkr46: FailedScheduling - no nodes available to schedule pods 2026/07/05 17:42:46 17:39:07 WARN Pod/dashboard-redirect-7bf9c448b9-msvwd: FailedScheduling - no nodes available to schedule pods 2026/07/05 17:42:46 17:39:07 WARN Pod/rhods-dashboard-5fd8c7f95b-9svc7: FailedScheduling - no nodes available to schedule pods 2026/07/05 17:42:46 17:39:05 WARN Pod/mlflow-operator-controller-manager-75654dffb8-wkr46: FailedScheduling - no nodes available to schedule pods 2026/07/05 17:42:46 17:39:05 WARN Pod/dashboard-redirect-7bf9c448b9-msvwd: FailedScheduling - no nodes available to schedule pods 2026/07/05 17:42:46 17:39:05 WARN Pod/rhods-dashboard-5fd8c7f95b-9svc7: FailedScheduling - no nodes available to schedule pods 2026/07/05 17:42:46 17:38:37 WARN Pod/postgres-69fb4594cf-pclwj: FailedScheduling - no nodes available to schedule pods 2026/07/05 17:42:46 17:38:37 WARN Pod/trustyai-service-operator-controller-manager-7bcdbcb4fd-xs5lx: FailedScheduling - no nodes available to schedule pods 2026/07/05 17:42:46 17:38:35 WARN Pod/postgres-69fb4594cf-pclwj: FailedScheduling - no nodes available to schedule pods 2026/07/05 17:42:46 17:38:35 WARN Pod/trustyai-service-operator-controller-manager-7bcdbcb4fd-xs5lx: FailedScheduling - no nodes available to schedule pods 2026/07/05 17:42:46 === RESOURCE QUOTAS === 2026/07/05 17:42:46 No resource quotas found in monitored namespaces 2026/07/05 17:42:46 Classification for group 3: infrastructure/pod-startup (code=1002, confidence=high) 2026/07/05 17:42:46 Evidence: pod dashboard-redirect-7bf9c448b9-msvwd stuck in Pending for 8m56s FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["pod dashboard-redirect-7bf9c448b9-msvwd stuck in Pending for 8m56s"],"confidence":"high"} 2026/07/05 17:42:46 Diagnostics complete for failed test: group 3 controller_test.go:763: Stopping: group 3 test failed. 2026/07/05 17:42:46 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/05 17:42:46 Circuit breaker: infrastructure/unknown failure recorded (3/3) 2026/07/05 17:42:46 Circuit breaker: threshold reached (3 failures), running confirming health check... 2026/07/05 17:42:46 CIRCUIT BREAKER TRIPPED: Infrastructure failure detected after 3 infrastructure/unknown failures. Cluster health issues: [operator deployment rhods-operator: 0/3 ready; operator pod rhods-operator-86ccbf48b5-2kszh: Pending; operator pod rhods-operator-86ccbf48b5-hcm7c: Pending; dependent kueue: deployment 0/2 ready; pod openshift-kueue-operator-8568d5f8-z45rd: Pending; dependent jobset: deployment 0/1 ready; pod jobset-operator-56d489b4b8-nbshb: Pending; dependent leader-worker-set: deployment 0/1 ready; dependent cert-manager: deployment 0/1 ready; dependent kuadrant: deployment 0/1 ready; pod authorino-operator-6d85f6564-tbbgz: Pending] --- FAIL: TestOdhOperator/components (1394.71s) === RUN TestOdhOperator 2026/07/05 17:14:20 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/05 17:19:31 TEST FAILURE DETECTED: DSCInitialization and DataScienceCluster validation E2E Tests 2026/07/05 17:19:31 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:DSCInitialization and DataScienceCluster validation E2E Tests) === 2026/07/05 17:19:31 === CLUSTER STATE === 2026/07/05 17:19:31 Node: ip-10-0-130-40.ec2.internal 2026/07/05 17:19:31 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/05 17:19:31 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/05 17:19:31 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/05 17:19:31 Condition Ready: True - kubelet is posting ready status 2026/07/05 17:19:31 Resources: 7500m CPU, 30892168Ki memory allocatable (of 8000m CPU, 32043144Ki memory capacity) 2026/07/05 17:19:31 Node: ip-10-0-131-27.ec2.internal 2026/07/05 17:19:31 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/05 17:19:31 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/05 17:19:31 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/05 17:19:31 Condition Ready: True - kubelet is posting ready status 2026/07/05 17:19:31 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/05 17:19:31 Node: ip-10-0-133-110.ec2.internal 2026/07/05 17:19:31 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/05 17:19:31 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/05 17:19:31 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/05 17:19:31 Condition Ready: True - kubelet is posting ready status 2026/07/05 17:19:31 Resources: 7500m CPU, 31236228Ki memory allocatable (of 8000m CPU, 32387204Ki memory capacity) 2026/07/05 17:19:31 === DEPLOYMENTS === 2026/07/05 17:19:31 Namespace: redhat-ods-applications 2026/07/05 17:19:31 Deployment dashboard-redirect: 2/2 (READY) 2026/07/05 17:19:31 Deployment data-science-pipelines-operator-controller-manager: 1/1 (READY) 2026/07/05 17:19:31 Deployment feast-operator-controller-manager: 1/1 (READY) 2026/07/05 17:19:31 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/05 17:19:31 Deployment kubeflow-trainer-controller-manager: 1/1 (READY) 2026/07/05 17:19:31 Deployment kubeflow-training-operator: 1/1 (READY) 2026/07/05 17:19:31 Deployment kuberay-operator: 1/1 (READY) 2026/07/05 17:19:31 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/05 17:19:31 Deployment maas-api: 1/1 (READY) 2026/07/05 17:19:31 Deployment maas-controller: 1/1 (READY) 2026/07/05 17:19:31 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/05 17:19:31 Deployment model-registry-operator-controller-manager: 1/1 (READY) 2026/07/05 17:19:31 Deployment model-serving-api: 1/1 (READY) 2026/07/05 17:19:31 Deployment notebook-controller-deployment: 1/1 (READY) 2026/07/05 17:19:31 Deployment odh-model-controller: 1/1 (READY) 2026/07/05 17:19:31 Deployment odh-notebook-controller-manager: 1/1 (READY) 2026/07/05 17:19:31 Deployment ogx-k8s-operator-controller-manager: 1/1 (READY) 2026/07/05 17:19:31 Deployment postgres: 1/1 (READY) 2026/07/05 17:19:31 Deployment rhods-dashboard: 2/2 (READY) 2026/07/05 17:19:31 Deployment spark-operator-controller: 1/1 (READY) 2026/07/05 17:19:31 Deployment spark-operator-webhook: 1/1 (READY) 2026/07/05 17:19:31 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/05 17:19:31 Namespace: kube-system 2026/07/05 17:19:31 === PODS === 2026/07/05 17:19:31 Namespace: redhat-ods-applications 2026/07/05 17:19:31 Pod maas-api-key-cleanup-29721165-tfxqm: Succeeded 2026/07/05 17:19:31 Container cleanup: ready=false, terminated: Completed (exit 0) 2026/07/05 17:19:31 Pod maas-api-key-cleanup-29721180-h56jn: Succeeded 2026/07/05 17:19:31 Container cleanup: ready=false, terminated: Completed (exit 0) 2026/07/05 17:19:31 Pod maas-api-key-cleanup-29721195-z2n7l: Succeeded 2026/07/05 17:19:31 Container cleanup: ready=false, terminated: Completed (exit 0) 2026/07/05 17:19:31 Namespace kube-system: no problematic pods found 2026/07/05 17:19:31 === LOGS (CURRENT) for container cleanup === 2026/07/05 17:19:31 {"deletedCount":0,"message":"Successfully deleted 0 expired ephemeral key(s)"} 2026/07/05 17:19:31 === END LOGS === 2026/07/05 17:19:31 === LOGS (CURRENT) for container cleanup === 2026/07/05 17:19:31 {"deletedCount":0,"message":"Successfully deleted 0 expired ephemeral key(s)"} 2026/07/05 17:19:31 === END LOGS === 2026/07/05 17:19:31 === LOGS (CURRENT) for container cleanup === 2026/07/05 17:19:31 {"deletedCount":0,"message":"Successfully deleted 0 expired ephemeral key(s)"} 2026/07/05 17:19:31 === END LOGS === 2026/07/05 17:19:31 === OPERATOR STATUS === 2026/07/05 17:19:31 Operator deployment: 3/3 ready 2026/07/05 17:19:31 === DSCI STATUS === 2026/07/05 17:19:31 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/05 17:19:31 === DSC STATUS === 2026/07/05 17:19:31 DSC default-dsc: 2026/07/05 17:19:31 Ready: True - 2026/07/05 17:19:31 ProvisioningSucceeded: True - 2026/07/05 17:19:31 ModulesReady: True - All registered modules have ManagementState Removed or are not configured 2026/07/05 17:19:31 AIPipelinesReady: True - 2026/07/05 17:19:31 ComponentsReady: True - 2026/07/05 17:19:31 DashboardReady: True - 2026/07/05 17:19:31 FeastOperatorReady: True - 2026/07/05 17:19:31 KserveLLMInferenceServiceDependencies: True - 2026/07/05 17:19:31 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/05 17:19:31 KserveReady: True - 2026/07/05 17:19:31 KueueReady: False - Component ManagementState is set to Removed 2026/07/05 17:19:31 LLM-D-WVADependencies: True - 2026/07/05 17:19:31 MLflowOperatorReady: True - 2026/07/05 17:19:31 MaaSPrerequisitesAvailable: True - MaaS gateway annotations and Authorino TLS are correctly configured 2026/07/05 17:19:31 ModelControllerReady: True - 2026/07/05 17:19:31 ModelRegistryReady: True - 2026/07/05 17:19:31 ModelsAsServiceReady: True - MaaS platform manifests applied and maas-api deployment is available 2026/07/05 17:19:31 OGXReady: True - 2026/07/05 17:19:31 RayReady: True - 2026/07/05 17:19:31 SparkOperatorReady: True - 2026/07/05 17:19:31 TrainerReady: True - 2026/07/05 17:19:31 TrainingOperatorReady: True - 2026/07/05 17:19:31 TrustyAIReady: True - 2026/07/05 17:19:31 WorkbenchesReady: True - 2026/07/05 17:19:31 === RECENT EVENTS === 2026/07/05 17:19:31 17:15:04 INFO Job/maas-api-key-cleanup-29721195: Completed - Job completed 2026/07/05 17:19:31 17:15:04 INFO CronJob/maas-api-key-cleanup: SuccessfulDelete - Deleted job maas-api-key-cleanup-29721150 2026/07/05 17:19:31 17:15:04 INFO CronJob/maas-api-key-cleanup: SawCompletedJob - Saw completed job: maas-api-key-cleanup-29721195, condition: Complete 2026/07/05 17:19:31 17:15:01 INFO Pod/maas-api-key-cleanup-29721195-z2n7l: Created - Created container: cleanup 2026/07/05 17:19:31 17:15:01 INFO Pod/maas-api-key-cleanup-29721195-z2n7l: Started - Started container cleanup 2026/07/05 17:19:31 17:15:00 INFO Pod/maas-api-key-cleanup-29721195-z2n7l: Scheduled - Successfully assigned redhat-ods-applications/maas-api-key-cleanup-29721195-z2n7l to ip-10-0-130-40.ec2.internal 2026/07/05 17:19:31 17:15:00 INFO Pod/maas-api-key-cleanup-29721195-z2n7l: AddedInterface - Add eth0 [10.133.0.75/23] from ovn-kubernetes 2026/07/05 17:19:31 17:15:00 INFO Pod/maas-api-key-cleanup-29721195-z2n7l: Pulled - Container image "registry.redhat.io/ubi9/ubi-minimal@sha256:907b68736aa798b2d38255b7aa070b2a70acb90803864a40f05d0ec47556ddd0" already present on machine 2026/07/05 17:19:31 17:15:00 INFO Job/maas-api-key-cleanup-29721195: SuccessfulCreate - Created pod: maas-api-key-cleanup-29721195-z2n7l 2026/07/05 17:19:31 17:15:00 INFO CronJob/maas-api-key-cleanup: SuccessfulCreate - Created job maas-api-key-cleanup-29721195 2026/07/05 17:19:31 === RESOURCE QUOTAS === 2026/07/05 17:19:31 No resource quotas found in monitored namespaces 2026/07/05 17:19:31 Classification for DSCInitialization and DataScienceCluster validation E2E Tests: infrastructure/dsci-unhealthy (code=1009, confidence=medium) 2026/07/05 17:19:31 Evidence: DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"dsci-unhealthy","error_code":1009,"evidence":["DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False"],"confidence":"medium"} 2026/07/05 17:19:31 Diagnostics complete for failed test: DSCInitialization and DataScienceCluster validation E2E Tests controller_test.go:478: Stopping: DSCInitialization and DataScienceCluster validation E2E Tests test failed. 2026/07/05 17:19:31 Circuit breaker: failure classified as infrastructure/dsci-unhealthy (code=1009) 2026/07/05 17:19:31 Circuit breaker: infrastructure/unknown failure recorded (1/3) 2026/07/05 17:42:46 TEST FAILURE DETECTED: components 2026/07/05 17:42:46 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/05 17:42:46 === CLUSTER STATE === 2026/07/05 17:42:46 No nodes found in cluster 2026/07/05 17:42:46 === DEPLOYMENTS === 2026/07/05 17:42:46 Failed to collect deployment data: deployments not ready: redhat-ods-applications/dashboard-redirect (0/2), redhat-ods-applications/mlflow-operator-controller-manager (0/1), redhat-ods-applications/postgres (0/1), redhat-ods-applications/postgres-deployment (0/1), redhat-ods-applications/rhods-dashboard (0/2), redhat-ods-applications/trustyai-service-operator-controller-manager (0/1) 2026/07/05 17:42:46 === PODS === 2026/07/05 17:42:46 Failed to collect pod data: redhat-ods-applications/dashboard-redirect-7bf9c448b9-msvwd: phase=Pending; redhat-ods-applications/mlflow-operator-controller-manager-75654dffb8-wkr46: phase=Pending; redhat-ods-applications/postgres-69fb4594cf-pclwj: phase=Pending; redhat-ods-applications/rhods-dashboard-5fd8c7f95b-9svc7: phase=Pending; redhat-ods-applications/trustyai-service-operator-controller-manager-7bcdbcb4fd-xs5lx: phase=Pending 2026/07/05 17:42:46 === OPERATOR STATUS === 2026/07/05 17:42:46 Failed to collect operator data: operator deployment rhods-operator: 0/3 ready; operator pod rhods-operator-86ccbf48b5-2kszh: Pending; operator pod rhods-operator-86ccbf48b5-hcm7c: Pending; dependent kueue: deployment 0/2 ready; pod openshift-kueue-operator-8568d5f8-z45rd: Pending; dependent jobset: deployment 0/1 ready; pod jobset-operator-56d489b4b8-nbshb: Pending; dependent leader-worker-set: deployment 0/1 ready; dependent cert-manager: deployment 0/1 ready; dependent kuadrant: deployment 0/1 ready; pod authorino-operator-6d85f6564-tbbgz: Pending 2026/07/05 17:42:46 === DSCI STATUS === 2026/07/05 17:42:46 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/05 17:42:46 === DSC STATUS === 2026/07/05 17:42:46 Failed to collect DSC data: default-dsc unhealthy conditions: ModelsAsServiceReady=False 2026/07/05 17:42:46 === RECENT EVENTS === 2026/07/05 17:42:46 17:41:49 WARN Job/mlflow-mg-3130-g1: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/05 17:42:46 17:40:48 WARN ReplicaSet/dashboard-redirect-7bf9c448b9: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/05 17:42:46 17:40:48 WARN ReplicaSet/rhods-dashboard-5fd8c7f95b: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/05 17:42:46 17:40:48 WARN ReplicaSet/postgres-deployment-86754cb759: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/05 17:42:46 17:39:07 WARN Pod/mlflow-operator-controller-manager-75654dffb8-wkr46: FailedScheduling - no nodes available to schedule pods 2026/07/05 17:42:46 17:39:07 WARN Pod/dashboard-redirect-7bf9c448b9-msvwd: FailedScheduling - no nodes available to schedule pods 2026/07/05 17:42:46 17:39:07 WARN Pod/rhods-dashboard-5fd8c7f95b-9svc7: FailedScheduling - no nodes available to schedule pods 2026/07/05 17:42:46 17:39:05 WARN Pod/mlflow-operator-controller-manager-75654dffb8-wkr46: FailedScheduling - no nodes available to schedule pods 2026/07/05 17:42:46 17:39:05 WARN Pod/dashboard-redirect-7bf9c448b9-msvwd: FailedScheduling - no nodes available to schedule pods 2026/07/05 17:42:46 17:39:05 WARN Pod/rhods-dashboard-5fd8c7f95b-9svc7: FailedScheduling - no nodes available to schedule pods 2026/07/05 17:42:46 17:38:37 WARN Pod/postgres-69fb4594cf-pclwj: FailedScheduling - no nodes available to schedule pods 2026/07/05 17:42:46 17:38:37 WARN Pod/trustyai-service-operator-controller-manager-7bcdbcb4fd-xs5lx: FailedScheduling - no nodes available to schedule pods 2026/07/05 17:42:46 17:38:35 WARN Pod/postgres-69fb4594cf-pclwj: FailedScheduling - no nodes available to schedule pods 2026/07/05 17:42:46 17:38:35 WARN Pod/trustyai-service-operator-controller-manager-7bcdbcb4fd-xs5lx: FailedScheduling - no nodes available to schedule pods 2026/07/05 17:42:46 === RESOURCE QUOTAS === 2026/07/05 17:42:46 No resource quotas found in monitored namespaces 2026/07/05 17:42:46 Classification for components: infrastructure/pod-startup (code=1002, confidence=high) 2026/07/05 17:42:46 Evidence: pod dashboard-redirect-7bf9c448b9-msvwd stuck in Pending for 8m56s FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["pod dashboard-redirect-7bf9c448b9-msvwd stuck in Pending for 8m56s"],"confidence":"high"} 2026/07/05 17:42:46 Diagnostics complete for failed test: components controller_test.go:491: Stopping: components test failed. Deletion Policy: Never. Skipping deletion tests. 2026/07/05 17:42:46 === CIRCUIT BREAKER SUMMARY === 2026/07/05 17:42:46 Tripped 1 time(s) 2026/07/05 17:42:46 Reason: Infrastructure failure detected after 3 infrastructure/unknown failures. Cluster health issues: [operator deployment rhods-operator: 0/3 ready; operator pod rhods-operator-86ccbf48b5-2kszh: Pending; operator pod rhods-operator-86ccbf48b5-hcm7c: Pending; dependent kueue: deployment 0/2 ready; pod openshift-kueue-operator-8568d5f8-z45rd: Pending; dependent jobset: deployment 0/1 ready; pod jobset-operator-56d489b4b8-nbshb: Pending; dependent leader-worker-set: deployment 0/1 ready; dependent cert-manager: deployment 0/1 ready; dependent kuadrant: deployment 0/1 ready; pod authorino-operator-6d85f6564-tbbgz: Pending] 2026/07/05 17:42:46 =============================== controller_test.go:433: Circuit breaker tripped: Infrastructure failure detected after 3 infrastructure/unknown failures. Cluster health issues: [operator deployment rhods-operator: 0/3 ready; operator pod rhods-operator-86ccbf48b5-2kszh: Pending; operator pod rhods-operator-86ccbf48b5-hcm7c: Pending; dependent kueue: deployment 0/2 ready; pod openshift-kueue-operator-8568d5f8-z45rd: Pending; dependent jobset: deployment 0/1 ready; pod jobset-operator-56d489b4b8-nbshb: Pending; dependent leader-worker-set: deployment 0/1 ready; dependent cert-manager: deployment 0/1 ready; dependent kuadrant: deployment 0/1 ready; pod authorino-operator-6d85f6564-tbbgz: Pending] --- FAIL: TestOdhOperator (1705.51s) === RUN TestRunExcluding/runs_all_except_excluded testgroup_test.go:119: Error Trace: /workspace/tests/e2e/testgroup_test.go:119 Error: []string{} does not contain "auth" Test: TestRunExcluding/runs_all_except_excluded --- FAIL: TestRunExcluding/runs_all_except_excluded (0.00s) === RUN TestRunExcluding --- FAIL: TestRunExcluding (0.00s)