[FAILED] Failed to create experiment Unexpected error: <*util.UserError | 0x2a69ad38a780>: Failed to create experiment. Params: '&{Experiment:0x2a69aaf37a80 timeout:30000000000 Context:context.Background.WithDeadline(2026-07-03 11:57:20.075659831 +0000 UTC m=+105.693395475 [34.994338339s]) HTTPClient:<nil>}'. Body: '&{CreatedAt:0001-01-01T00:00:00.000Z Description: DisplayName:E2EExperiment-1783079805075657051 ExperimentID: LastRunCreatedAt:0001-01-01T00:00:00.000Z Namespace:dspa-test-l5rgxfei StorageState:<nil>}': Post "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments": EOF { internalError: <*errors.withStack | 0x2a69ab23c360>{ error: <*errors.withMessage | 0x2a69aa530360>{ cause: <*url.Error | 0x2a69ad38a720>{ Op: "Post", URL: "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments", Err: <*errors.errorString | 0x5c15e50>{s: "EOF"}, }, msg: "Failed to create experiment. Params: '&{Experiment:0x2a69aaf37a80 timeout:30000000000 Context:context.Background.WithDeadline(2026-07-03 11:57:20.075659831 +0000 UTC m=+105.693395475 [34.994338339s]) HTTPClient:<nil>}'. Body: '&{CreatedAt:0001-01-01T00:00:00.000Z Description: DisplayName:E2EExperiment-1783079805075657051 ExperimentID: LastRunCreatedAt:0001-01-01T00:00:00.000Z Namespace:dspa-test-l5rgxfei StorageState:<nil>}'", }, stack: [0x1a01bf8, 0x2a0c92c, 0x2b5bdcc, 0x2b6835c, 0x2b1a153, 0x2b33a99, 0x49a241], }, externalMessage: "Failed to create experiment 'E2EExperiment-1783079805075657051'. Raw error from the service: Post \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments\": EOF", externalStatusCode: 13, } occurred In [BeforeEach] at: /dspa/backend/test/testutil/experiment_utils.go:48 @ 07/03/26 11:56:45.081 There were additional failures detected after the initial failure. These are visible in the timeline > Enter [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/03/26 11:55:34.758 Create an experiment with name E2EExperiment-1783079734758010032 < Exit [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/03/26 11:55:34.915 (158ms) > Enter [BeforeEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:49 @ 07/03/26 11:55:34.915 ################### Setup before each Pipeline Upload test ##################### ################### Global Setup before each test ##################### Test Context: 0x2a69aa5c1c80 Looks like test-secret-1 already exists, because creation failed due to secrets "test-secret-1" already exists Looks like test-secret-2 already exists, because creation failed due to secrets "test-secret-2" already exists Looks like test-secret-3 already exists, because creation failed due to secrets "test-secret-3" already exists < Exit [BeforeEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:49 @ 07/03/26 11:55:34.964 (48ms) > Enter [It] Upload critical/flip_coin.yaml pipeline - /dspa/backend/test/end2end/pipeline_e2e_test.go:152 @ 07/03/26 11:55:34.964 Uploading pipeline file critical/flip_coin.yaml Creating temp pipeline file with overridden SDK Version Uploading pipeline with name=e2e-test-1783079734915866409, from file /dspa/test_data/sdk_compiled_pipelines/valid/critical/flip_coin.yaml Upload of pipeline file 'critical/flip_coin.yaml' successful Listing pipeline versions for pipeline e90c6ef2-35cd-4b3c-847a-cc1d0513c945 Get the pipeline run time inputs from pipeline spec file /dspa/test_data/sdk_compiled_pipelines/valid/critical/flip_coin.yaml Returning pipeline run time inputs map[] Create run for pipeline with id: 'e90c6ef2-35cd-4b3c-847a-cc1d0513c945' and name: 'e2e-test-1783079734915866409' Create a pipeline run for pipeline with id=e90c6ef2-35cd-4b3c-847a-cc1d0513c945 and versionId=fd0c30dc-7cf9-4e6f-9d6d-a5a611aaf919 Create a pipeline run body Created Pipeline Run successfully with runId=f145d48c-a0dd-4a25-abd3-2488df20a134 Created Pipeline Run with id: f145d48c-a0dd-4a25-abd3-2488df20a134 for pipeline with id: e90c6ef2-35cd-4b3c-847a-cc1d0513c945 Waiting for pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 to be in one of '[SUCCEEDED SKIPPED FAILED CANCELED]' Get a pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 Pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 is in PENDING state, waiting... Get a pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 Pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 is in RUNNING state, waiting... Get a pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 Pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 is in RUNNING state, waiting... Get a pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 Pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 is in RUNNING state, waiting... Get a pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 Pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 is in RUNNING state, waiting... Get a pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 Pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 is in RUNNING state, waiting... Get a pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 Pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 is in RUNNING state, waiting... Get a pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 Pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 is in RUNNING state, waiting... Get a pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 Pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 is in RUNNING state, waiting... Get a pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 Pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 is in RUNNING state, waiting... Get a pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 Pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 is in RUNNING state, waiting... Get a pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 Pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 is in RUNNING state, waiting... Get a pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 [FAILED] Failure recorded during attempt 1: Failed to get run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 Unexpected error: <*util.UserError | 0x2a69ab28c630>: Failed to get run. Params: '&{ExperimentID:<nil> RunID:f145d48c-a0dd-4a25-abd3-2488df20a134 timeout:0 Context:context.Background.WithDeadline(2026-07-03 11:57:10.458165034 +0000 UTC m=+96.075900688 [26.825388294s]) HTTPClient:<nil>}': issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Get "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/runs/f145d48c-a0dd-4a25-abd3-2488df20a134": EOF { internalError: <*errors.withStack | 0x2a69ab15c450>{ error: <*errors.withMessage | 0x2a69aa80e9e0>{ cause: <*errors.errorString | 0x2a69ab153ea0>{ s: "issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Get \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/runs/f145d48c-a0dd-4a25-abd3-2488df20a134\": EOF", }, msg: "Failed to get run. Params: '&{ExperimentID:<nil> RunID:f145d48c-a0dd-4a25-abd3-2488df20a134 timeout:0 Context:context.Background.WithDeadline(2026-07-03 11:57:10.458165034 +0000 UTC m=+96.075900688 [26.825388294s]) HTTPClient:<nil>}'", }, stack: [0x1a01bf8, 0x2a124bd, 0x2b5f09b, 0x2b5f30c, 0x2b622ec, 0x2b74098, 0x2b72a2a, 0x2b1a153, 0x2b33a99, 0x49a241], }, externalMessage: "Failed to get run 'f145d48c-a0dd-4a25-abd3-2488df20a134'. Raw error from the service: issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Get \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/runs/f145d48c-a0dd-4a25-abd3-2488df20a134\": EOF", externalStatusCode: 13, } occurred In [It] at: /dspa/backend/test/testutil/pipeline_run_utils.go:82 @ 07/03/26 11:56:43.633 < Exit [It] Upload critical/flip_coin.yaml pipeline - /dspa/backend/test/end2end/pipeline_e2e_test.go:152 @ 07/03/26 11:56:43.633 (1m8.669s) > Enter [DeferCleanup (Each)] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:96 @ 07/03/26 11:56:43.633 ################### Global Cleanup after each test ##################### Deleting 1 run(s) Skipping Termination of the run f145d48c-a0dd-4a25-abd3-2488df20a134, because it does not exist Skipping Archiving of the run f145d48c-a0dd-4a25-abd3-2488df20a134, because it does not exist Skipping Deletion of the run f145d48c-a0dd-4a25-abd3-2488df20a134, as it does not exist Deleting 0 experiment(s) Deleting 1 pipeline(s) Deleting pipeline with id=e90c6ef2-35cd-4b3c-847a-cc1d0513c945 (cascade=true) [FAILED] Error occurred while deleting pipeline with id=e90c6ef2-35cd-4b3c-847a-cc1d0513c945 Unexpected error: <*util.UserError | 0x2a69ab28dc80>: Failed to delete pipeline. Params: '&{Cascade:0x2a69ab15f0eb PipelineID:e90c6ef2-35cd-4b3c-847a-cc1d0513c945 timeout:0 Context:context.Background.WithDeadline(2026-07-03 11:57:20.069341899 +0000 UTC m=+105.687077553 [34.99433386s]) HTTPClient:<nil>}': issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Delete "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/pipelines/e90c6ef2-35cd-4b3c-847a-cc1d0513c945?cascade=true": EOF { internalError: <*errors.withStack | 0x2a69ab15c828>{ error: <*errors.withMessage | 0x2a69aa80f8a0>{ cause: <*errors.errorString | 0x2a69ab298ac0>{ s: "issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Delete \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/pipelines/e90c6ef2-35cd-4b3c-847a-cc1d0513c945?cascade=true\": EOF", }, msg: "Failed to delete pipeline. Params: '&{Cascade:0x2a69ab15f0eb PipelineID:e90c6ef2-35cd-4b3c-847a-cc1d0513c945 timeout:0 Context:context.Background.WithDeadline(2026-07-03 11:57:20.069341899 +0000 UTC m=+105.687077553 [34.99433386s]) HTTPClient:<nil>}'", }, stack: [0x1a01bf8, 0x2a0d673, 0x2b60255, 0x2b74532, 0x2b742a9, 0x511506, 0x5103b9, 0x2b1bf54, 0x2b1a153, 0x2b33a99, 0x49a241], }, externalMessage: "Failed to delete pipeline 'e90c6ef2-35cd-4b3c-847a-cc1d0513c945'. Raw error from the service: issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Delete \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/pipelines/e90c6ef2-35cd-4b3c-847a-cc1d0513c945?cascade=true\": EOF", externalStatusCode: 13, } occurred In [DeferCleanup (Each)] at: /dspa/backend/test/end2end/pipeline_e2e_test.go:331 @ 07/03/26 11:56:45.075 < Exit [DeferCleanup (Each)] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:96 @ 07/03/26 11:56:45.075 (1.442s) Attempt #1 Failed. Retrying ↺ @ 07/03/26 11:56:45.075 > Enter [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/03/26 11:56:45.075 Create an experiment with name E2EExperiment-1783079805075657051 [FAILED] Failed to create experiment Unexpected error: <*util.UserError | 0x2a69ad38a780>: Failed to create experiment. Params: '&{Experiment:0x2a69aaf37a80 timeout:30000000000 Context:context.Background.WithDeadline(2026-07-03 11:57:20.075659831 +0000 UTC m=+105.693395475 [34.994338339s]) HTTPClient:<nil>}'. Body: '&{CreatedAt:0001-01-01T00:00:00.000Z Description: DisplayName:E2EExperiment-1783079805075657051 ExperimentID: LastRunCreatedAt:0001-01-01T00:00:00.000Z Namespace:dspa-test-l5rgxfei StorageState:<nil>}': Post "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments": EOF { internalError: <*errors.withStack | 0x2a69ab23c360>{ error: <*errors.withMessage | 0x2a69aa530360>{ cause: <*url.Error | 0x2a69ad38a720>{ Op: "Post", URL: "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments", Err: <*errors.errorString | 0x5c15e50>{s: "EOF"}, }, msg: "Failed to create experiment. Params: '&{Experiment:0x2a69aaf37a80 timeout:30000000000 Context:context.Background.WithDeadline(2026-07-03 11:57:20.075659831 +0000 UTC m=+105.693395475 [34.994338339s]) HTTPClient:<nil>}'. Body: '&{CreatedAt:0001-01-01T00:00:00.000Z Description: DisplayName:E2EExperiment-1783079805075657051 ExperimentID: LastRunCreatedAt:0001-01-01T00:00:00.000Z Namespace:dspa-test-l5rgxfei StorageState:<nil>}'", }, stack: [0x1a01bf8, 0x2a0c92c, 0x2b5bdcc, 0x2b6835c, 0x2b1a153, 0x2b33a99, 0x49a241], }, externalMessage: "Failed to create experiment 'E2EExperiment-1783079805075657051'. Raw error from the service: Post \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments\": EOF", externalStatusCode: 13, } occurred In [BeforeEach] at: /dspa/backend/test/testutil/experiment_utils.go:48 @ 07/03/26 11:56:45.081 < Exit [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/03/26 11:56:45.081 (6ms) > Enter [ReportAfterEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:99 @ 07/03/26 11:56:45.081 Argo Workflows archived logs - /dspa/backend/test/end2end/pipeline_e2e_test.go:105 @ 07/03/26 11:56:45.09 WARNING: failed to read mlpipeline-minio-artifact secret (dspa-test-l5rgxfei/mlpipeline-minio-artifact): secrets "mlpipeline-minio-artifact" not found < Exit [ReportAfterEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:99 @ 07/03/26 11:56:45.09 (9ms) > Enter [ReportAfterEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:155 @ 07/03/26 11:56:45.09 Test failed... Capturing logs Test Log - /dspa/backend/test/end2end/e2e_suite_test.go:158 @ 07/03/26 11:56:45.09 Create an experiment with name E2EExperiment-1783079734758010032 ################### Setup before each Pipeline Upload test ##################### ################### Global Setup before each test ##################### Test Context: 0x2a69aa5c1c80 Looks like test-secret-1 already exists, because creation failed due to secrets "test-secret-1" already exists Looks like test-secret-2 already exists, because creation failed due to secrets "test-secret-2" already exists Looks like test-secret-3 already exists, because creation failed due to secrets "test-secret-3" already exists Uploading pipeline file critical/flip_coin.yaml Creating temp pipeline file with overridden SDK Version Uploading pipeline with name=e2e-test-1783079734915866409, from file /dspa/test_data/sdk_compiled_pipelines/valid/critical/flip_coin.yaml Upload of pipeline file 'critical/flip_coin.yaml' successful Listing pipeline versions for pipeline e90c6ef2-35cd-4b3c-847a-cc1d0513c945 Get the pipeline run time inputs from pipeline spec file /dspa/test_data/sdk_compiled_pipelines/valid/critical/flip_coin.yaml Returning pipeline run time inputs map[] Create run for pipeline with id: 'e90c6ef2-35cd-4b3c-847a-cc1d0513c945' and name: 'e2e-test-1783079734915866409' Create a pipeline run for pipeline with id=e90c6ef2-35cd-4b3c-847a-cc1d0513c945 and versionId=fd0c30dc-7cf9-4e6f-9d6d-a5a611aaf919 Create a pipeline run body Created Pipeline Run successfully with runId=f145d48c-a0dd-4a25-abd3-2488df20a134 Created Pipeline Run with id: f145d48c-a0dd-4a25-abd3-2488df20a134 for pipeline with id: e90c6ef2-35cd-4b3c-847a-cc1d0513c945 Waiting for pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 to be in one of '[SUCCEEDED SKIPPED FAILED CANCELED]' Get a pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 Pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 is in PENDING state, waiting... Get a pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 Pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 is in RUNNING state, waiting... Get a pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 Pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 is in RUNNING state, waiting... Get a pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 Pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 is in RUNNING state, waiting... Get a pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 Pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 is in RUNNING state, waiting... Get a pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 Pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 is in RUNNING state, waiting... Get a pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 Pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 is in RUNNING state, waiting... Get a pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 Pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 is in RUNNING state, waiting... Get a pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 Pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 is in RUNNING state, waiting... Get a pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 Pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 is in RUNNING state, waiting... Get a pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 Pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 is in RUNNING state, waiting... Get a pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 Pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 is in RUNNING state, waiting... Get a pipeline run with id=f145d48c-a0dd-4a25-abd3-2488df20a134 ################### Global Cleanup after each test ##################### Deleting 1 run(s) Skipping Termination of the run f145d48c-a0dd-4a25-abd3-2488df20a134, because it does not exist Skipping Archiving of the run f145d48c-a0dd-4a25-abd3-2488df20a134, because it does not exist Skipping Deletion of the run f145d48c-a0dd-4a25-abd3-2488df20a134, as it does not exist Deleting 0 experiment(s) Deleting 1 pipeline(s) Deleting pipeline with id=e90c6ef2-35cd-4b3c-847a-cc1d0513c945 (cascade=true) Create an experiment with name E2EExperiment-1783079805075657051 Failed to create log file due to: open /dspa/backend/test/end2end/logs/Upload and Verify Pipeline Run > Upload a pipeline file, run it and verify that pipeline run succeeds Smoke > Upload critical/flip_coin.yaml pipeline.log: no such file or directory Failed to write to the log file, due to: invalid argument < Exit [ReportAfterEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:155 @ 07/03/26 11:56:45.09 (0s) [FAILED] Failed to create experiment Unexpected error: <*util.UserError | 0x1c1a12fb9140>: Failed to create experiment. Params: '&{Experiment:0x1c1a12e4a800 timeout:30000000000 Context:context.Background.WithDeadline(2026-07-03 11:57:20.081885631 +0000 UTC m=+105.698711944 [34.992637421s]) HTTPClient:<nil>}'. Body: '&{CreatedAt:0001-01-01T00:00:00.000Z Description: DisplayName:E2EExperiment-1783079805081882491 ExperimentID: LastRunCreatedAt:0001-01-01T00:00:00.000Z Namespace:dspa-test-l5rgxfei StorageState:<nil>}': Post "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments": EOF { internalError: <*errors.withStack | 0x1c1a12d088b8>{ error: <*errors.withMessage | 0x1c1a125199e0>{ cause: <*url.Error | 0x1c1a12fb90e0>{ Op: "Post", URL: "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments", Err: <*errors.errorString | 0x5c15e50>{s: "EOF"}, }, msg: "Failed to create experiment. Params: '&{Experiment:0x1c1a12e4a800 timeout:30000000000 Context:context.Background.WithDeadline(2026-07-03 11:57:20.081885631 +0000 UTC m=+105.698711944 [34.992637421s]) HTTPClient:<nil>}'. Body: '&{CreatedAt:0001-01-01T00:00:00.000Z Description: DisplayName:E2EExperiment-1783079805081882491 ExperimentID: LastRunCreatedAt:0001-01-01T00:00:00.000Z Namespace:dspa-test-l5rgxfei StorageState:<nil>}'", }, stack: [0x1a01bf8, 0x2a0c92c, 0x2b5bdcc, 0x2b6835c, 0x2b1a153, 0x2b33a99, 0x49a241], }, externalMessage: "Failed to create experiment 'E2EExperiment-1783079805081882491'. Raw error from the service: Post \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments\": EOF", externalStatusCode: 13, } occurred In [BeforeEach] at: /dspa/backend/test/testutil/experiment_utils.go:48 @ 07/03/26 11:56:45.089 There were additional failures detected after the initial failure. These are visible in the timeline > Enter [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/03/26 11:55:34.757 Create an experiment with name E2EExperiment-1783079734757999362 < Exit [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/03/26 11:55:34.911 (153ms) > Enter [BeforeEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:49 @ 07/03/26 11:55:34.911 ################### Setup before each Pipeline Upload test ##################### ################### Global Setup before each test ##################### Test Context: 0x1c1a12c4a380 Looks like test-secret-1 already exists, because creation failed due to secrets "test-secret-1" already exists Looks like test-secret-2 already exists, because creation failed due to secrets "test-secret-2" already exists Looks like test-secret-3 already exists, because creation failed due to secrets "test-secret-3" already exists < Exit [BeforeEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:49 @ 07/03/26 11:55:34.957 (46ms) > Enter [It] Upload critical/parallel_for_after_dependency.yaml pipeline - /dspa/backend/test/end2end/pipeline_e2e_test.go:152 @ 07/03/26 11:55:34.957 Uploading pipeline file critical/parallel_for_after_dependency.yaml Creating temp pipeline file with overridden SDK Version Uploading pipeline with name=e2e-test-1783079734911587686, from file /dspa/test_data/sdk_compiled_pipelines/valid/critical/parallel_for_after_dependency.yaml Upload of pipeline file 'critical/parallel_for_after_dependency.yaml' successful Listing pipeline versions for pipeline b5004880-ae57-4b69-a229-0a6283420a6e Get the pipeline run time inputs from pipeline spec file /dspa/test_data/sdk_compiled_pipelines/valid/critical/parallel_for_after_dependency.yaml Returning pipeline run time inputs map[] Create run for pipeline with id: 'b5004880-ae57-4b69-a229-0a6283420a6e' and name: 'e2e-test-1783079734911587686' Create a pipeline run for pipeline with id=b5004880-ae57-4b69-a229-0a6283420a6e and versionId=d8b248cd-238f-4c9d-bb6d-5db662082a54 Create a pipeline run body Created Pipeline Run successfully with runId=d217331a-76a2-4edd-9008-683abe2ef673 Created Pipeline Run with id: d217331a-76a2-4edd-9008-683abe2ef673 for pipeline with id: b5004880-ae57-4b69-a229-0a6283420a6e Waiting for pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 to be in one of '[SUCCEEDED SKIPPED FAILED CANCELED]' Get a pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 Pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 is in PENDING state, waiting... Get a pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 Pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 is in RUNNING state, waiting... Get a pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 Pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 is in RUNNING state, waiting... Get a pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 Pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 is in RUNNING state, waiting... Get a pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 Pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 is in RUNNING state, waiting... Get a pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 Pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 is in RUNNING state, waiting... Get a pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 Pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 is in RUNNING state, waiting... Get a pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 Pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 is in RUNNING state, waiting... Get a pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 Pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 is in RUNNING state, waiting... Get a pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 Pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 is in RUNNING state, waiting... Get a pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 Pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 is in RUNNING state, waiting... Get a pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 Pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 is in RUNNING state, waiting... Get a pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 [FAILED] Failure recorded during attempt 1: Failed to get run with id=d217331a-76a2-4edd-9008-683abe2ef673 Unexpected error: <*util.UserError | 0x1c1a12f30fc0>: Failed to get run. Params: '&{ExperimentID:<nil> RunID:d217331a-76a2-4edd-9008-683abe2ef673 timeout:0 Context:context.Background.WithDeadline(2026-07-03 11:57:10.314321927 +0000 UTC m=+95.931148239 [26.681555817s]) HTTPClient:<nil>}': issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Get "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/runs/d217331a-76a2-4edd-9008-683abe2ef673": EOF { internalError: <*errors.withStack | 0x1c1a1252fae8>{ error: <*errors.withMessage | 0x1c1a12729ea0>{ cause: <*errors.errorString | 0x1c1a12f8a0c0>{ s: "issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Get \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/runs/d217331a-76a2-4edd-9008-683abe2ef673\": EOF", }, msg: "Failed to get run. Params: '&{ExperimentID:<nil> RunID:d217331a-76a2-4edd-9008-683abe2ef673 timeout:0 Context:context.Background.WithDeadline(2026-07-03 11:57:10.314321927 +0000 UTC m=+95.931148239 [26.681555817s]) HTTPClient:<nil>}'", }, stack: [0x1a01bf8, 0x2a124bd, 0x2b5f09b, 0x2b5f30c, 0x2b622ec, 0x2b74098, 0x2b72a2a, 0x2b1a153, 0x2b33a99, 0x49a241], }, externalMessage: "Failed to get run 'd217331a-76a2-4edd-9008-683abe2ef673'. Raw error from the service: issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Get \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/runs/d217331a-76a2-4edd-9008-683abe2ef673\": EOF", externalStatusCode: 13, } occurred In [It] at: /dspa/backend/test/testutil/pipeline_run_utils.go:82 @ 07/03/26 11:56:43.633 < Exit [It] Upload critical/parallel_for_after_dependency.yaml pipeline - /dspa/backend/test/end2end/pipeline_e2e_test.go:152 @ 07/03/26 11:56:43.633 (1m8.675s) > Enter [DeferCleanup (Each)] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:96 @ 07/03/26 11:56:43.633 ################### Global Cleanup after each test ##################### Deleting 1 run(s) Skipping Termination of the run d217331a-76a2-4edd-9008-683abe2ef673, because it does not exist Skipping Archiving of the run d217331a-76a2-4edd-9008-683abe2ef673, because it does not exist Skipping Deletion of the run d217331a-76a2-4edd-9008-683abe2ef673, as it does not exist Deleting 0 experiment(s) Deleting 1 pipeline(s) Deleting pipeline with id=b5004880-ae57-4b69-a229-0a6283420a6e (cascade=true) [FAILED] Error occurred while deleting pipeline with id=b5004880-ae57-4b69-a229-0a6283420a6e Unexpected error: <*util.UserError | 0x1c1a12fb8c90>: Failed to delete pipeline. Params: '&{Cascade:0x1c1a12c0f63b PipelineID:b5004880-ae57-4b69-a229-0a6283420a6e timeout:0 Context:context.Background.WithDeadline(2026-07-03 11:57:20.074947467 +0000 UTC m=+105.691773781 [34.993643136s]) HTTPClient:<nil>}': issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Delete "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/pipelines/b5004880-ae57-4b69-a229-0a6283420a6e?cascade=true": EOF { internalError: <*errors.withStack | 0x1c1a12d08618>{ error: <*errors.withMessage | 0x1c1a125196c0>{ cause: <*errors.errorString | 0x1c1a12f8adf0>{ s: "issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Delete \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/pipelines/b5004880-ae57-4b69-a229-0a6283420a6e?cascade=true\": EOF", }, msg: "Failed to delete pipeline. Params: '&{Cascade:0x1c1a12c0f63b PipelineID:b5004880-ae57-4b69-a229-0a6283420a6e timeout:0 Context:context.Background.WithDeadline(2026-07-03 11:57:20.074947467 +0000 UTC m=+105.691773781 [34.993643136s]) HTTPClient:<nil>}'", }, stack: [0x1a01bf8, 0x2a0d673, 0x2b60255, 0x2b74532, 0x2b742a9, 0x511506, 0x5103b9, 0x2b1bf54, 0x2b1a153, 0x2b33a99, 0x49a241], }, externalMessage: "Failed to delete pipeline 'b5004880-ae57-4b69-a229-0a6283420a6e'. Raw error from the service: issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Delete \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/pipelines/b5004880-ae57-4b69-a229-0a6283420a6e?cascade=true\": EOF", externalStatusCode: 13, } occurred In [DeferCleanup (Each)] at: /dspa/backend/test/end2end/pipeline_e2e_test.go:331 @ 07/03/26 11:56:45.081 < Exit [DeferCleanup (Each)] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:96 @ 07/03/26 11:56:45.081 (1.448s) Attempt #1 Failed. Retrying ↺ @ 07/03/26 11:56:45.081 > Enter [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/03/26 11:56:45.081 Create an experiment with name E2EExperiment-1783079805081882491 [FAILED] Failed to create experiment Unexpected error: <*util.UserError | 0x1c1a12fb9140>: Failed to create experiment. Params: '&{Experiment:0x1c1a12e4a800 timeout:30000000000 Context:context.Background.WithDeadline(2026-07-03 11:57:20.081885631 +0000 UTC m=+105.698711944 [34.992637421s]) HTTPClient:<nil>}'. Body: '&{CreatedAt:0001-01-01T00:00:00.000Z Description: DisplayName:E2EExperiment-1783079805081882491 ExperimentID: LastRunCreatedAt:0001-01-01T00:00:00.000Z Namespace:dspa-test-l5rgxfei StorageState:<nil>}': Post "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments": EOF { internalError: <*errors.withStack | 0x1c1a12d088b8>{ error: <*errors.withMessage | 0x1c1a125199e0>{ cause: <*url.Error | 0x1c1a12fb90e0>{ Op: "Post", URL: "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments", Err: <*errors.errorString | 0x5c15e50>{s: "EOF"}, }, msg: "Failed to create experiment. Params: '&{Experiment:0x1c1a12e4a800 timeout:30000000000 Context:context.Background.WithDeadline(2026-07-03 11:57:20.081885631 +0000 UTC m=+105.698711944 [34.992637421s]) HTTPClient:<nil>}'. Body: '&{CreatedAt:0001-01-01T00:00:00.000Z Description: DisplayName:E2EExperiment-1783079805081882491 ExperimentID: LastRunCreatedAt:0001-01-01T00:00:00.000Z Namespace:dspa-test-l5rgxfei StorageState:<nil>}'", }, stack: [0x1a01bf8, 0x2a0c92c, 0x2b5bdcc, 0x2b6835c, 0x2b1a153, 0x2b33a99, 0x49a241], }, externalMessage: "Failed to create experiment 'E2EExperiment-1783079805081882491'. Raw error from the service: Post \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments\": EOF", externalStatusCode: 13, } occurred In [BeforeEach] at: /dspa/backend/test/testutil/experiment_utils.go:48 @ 07/03/26 11:56:45.089 < Exit [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/03/26 11:56:45.089 (8ms) > Enter [ReportAfterEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:99 @ 07/03/26 11:56:45.089 Argo Workflows archived logs - /dspa/backend/test/end2end/pipeline_e2e_test.go:105 @ 07/03/26 11:56:45.096 WARNING: failed to read mlpipeline-minio-artifact secret (dspa-test-l5rgxfei/mlpipeline-minio-artifact): secrets "mlpipeline-minio-artifact" not found < Exit [ReportAfterEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:99 @ 07/03/26 11:56:45.096 (7ms) > Enter [ReportAfterEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:155 @ 07/03/26 11:56:45.097 Test failed... Capturing logs Test Log - /dspa/backend/test/end2end/e2e_suite_test.go:158 @ 07/03/26 11:56:45.097 Create an experiment with name E2EExperiment-1783079734757999362 ################### Setup before each Pipeline Upload test ##################### ################### Global Setup before each test ##################### Test Context: 0x1c1a12c4a380 Looks like test-secret-1 already exists, because creation failed due to secrets "test-secret-1" already exists Looks like test-secret-2 already exists, because creation failed due to secrets "test-secret-2" already exists Looks like test-secret-3 already exists, because creation failed due to secrets "test-secret-3" already exists Uploading pipeline file critical/parallel_for_after_dependency.yaml Creating temp pipeline file with overridden SDK Version Uploading pipeline with name=e2e-test-1783079734911587686, from file /dspa/test_data/sdk_compiled_pipelines/valid/critical/parallel_for_after_dependency.yaml Upload of pipeline file 'critical/parallel_for_after_dependency.yaml' successful Listing pipeline versions for pipeline b5004880-ae57-4b69-a229-0a6283420a6e Get the pipeline run time inputs from pipeline spec file /dspa/test_data/sdk_compiled_pipelines/valid/critical/parallel_for_after_dependency.yaml Returning pipeline run time inputs map[] Create run for pipeline with id: 'b5004880-ae57-4b69-a229-0a6283420a6e' and name: 'e2e-test-1783079734911587686' Create a pipeline run for pipeline with id=b5004880-ae57-4b69-a229-0a6283420a6e and versionId=d8b248cd-238f-4c9d-bb6d-5db662082a54 Create a pipeline run body Created Pipeline Run successfully with runId=d217331a-76a2-4edd-9008-683abe2ef673 Created Pipeline Run with id: d217331a-76a2-4edd-9008-683abe2ef673 for pipeline with id: b5004880-ae57-4b69-a229-0a6283420a6e Waiting for pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 to be in one of '[SUCCEEDED SKIPPED FAILED CANCELED]' Get a pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 Pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 is in PENDING state, waiting... Get a pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 Pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 is in RUNNING state, waiting... Get a pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 Pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 is in RUNNING state, waiting... Get a pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 Pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 is in RUNNING state, waiting... Get a pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 Pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 is in RUNNING state, waiting... Get a pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 Pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 is in RUNNING state, waiting... Get a pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 Pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 is in RUNNING state, waiting... Get a pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 Pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 is in RUNNING state, waiting... Get a pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 Pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 is in RUNNING state, waiting... Get a pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 Pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 is in RUNNING state, waiting... Get a pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 Pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 is in RUNNING state, waiting... Get a pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 Pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 is in RUNNING state, waiting... Get a pipeline run with id=d217331a-76a2-4edd-9008-683abe2ef673 ################### Global Cleanup after each test ##################### Deleting 1 run(s) Skipping Termination of the run d217331a-76a2-4edd-9008-683abe2ef673, because it does not exist Skipping Archiving of the run d217331a-76a2-4edd-9008-683abe2ef673, because it does not exist Skipping Deletion of the run d217331a-76a2-4edd-9008-683abe2ef673, as it does not exist Deleting 0 experiment(s) Deleting 1 pipeline(s) Deleting pipeline with id=b5004880-ae57-4b69-a229-0a6283420a6e (cascade=true) Create an experiment with name E2EExperiment-1783079805081882491 Failed to create log file due to: open /dspa/backend/test/end2end/logs/Upload and Verify Pipeline Run > Upload a pipeline file, run it and verify that pipeline run succeeds Smoke > Upload critical/parallel_for_after_dependency.yaml pipeline.log: no such file or directory Failed to write to the log file, due to: invalid argument < Exit [ReportAfterEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:155 @ 07/03/26 11:56:45.097 (0s) [FAILED] Failed to create experiment Unexpected error: <*util.UserError | 0x268aca3ee3f0>: Failed to create experiment. Params: '&{Experiment:0x268ac9b7f580 timeout:30000000000 Context:context.Background.WithDeadline(2026-07-03 11:57:20.088292937 +0000 UTC m=+105.703083152 [34.994825836s]) HTTPClient:<nil>}'. Body: '&{CreatedAt:0001-01-01T00:00:00.000Z Description: DisplayName:E2EExperiment-1783079805088289777 ExperimentID: LastRunCreatedAt:0001-01-01T00:00:00.000Z Namespace:dspa-test-l5rgxfei StorageState:<nil>}': Post "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments": EOF { internalError: <*errors.withStack | 0x268ac96df908>{ error: <*errors.withMessage | 0x268aca0f7de0>{ cause: <*url.Error | 0x268aca3ee390>{ Op: "Post", URL: "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments", Err: <*errors.errorString | 0x5c15e50>{s: "EOF"}, }, msg: "Failed to create experiment. Params: '&{Experiment:0x268ac9b7f580 timeout:30000000000 Context:context.Background.WithDeadline(2026-07-03 11:57:20.088292937 +0000 UTC m=+105.703083152 [34.994825836s]) HTTPClient:<nil>}'. Body: '&{CreatedAt:0001-01-01T00:00:00.000Z Description: DisplayName:E2EExperiment-1783079805088289777 ExperimentID: LastRunCreatedAt:0001-01-01T00:00:00.000Z Namespace:dspa-test-l5rgxfei StorageState:<nil>}'", }, stack: [0x1a01bf8, 0x2a0c92c, 0x2b5bdcc, 0x2b6835c, 0x2b1a153, 0x2b33a99, 0x49a241], }, externalMessage: "Failed to create experiment 'E2EExperiment-1783079805088289777'. Raw error from the service: Post \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments\": EOF", externalStatusCode: 13, } occurred In [BeforeEach] at: /dspa/backend/test/testutil/experiment_utils.go:48 @ 07/03/26 11:56:45.093 There were additional failures detected after the initial failure. These are visible in the timeline > Enter [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/03/26 11:55:34.758 Create an experiment with name E2EExperiment-1783079734758596242 < Exit [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/03/26 11:55:34.911 (153ms) > Enter [BeforeEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:49 @ 07/03/26 11:55:34.911 ################### Setup before each Pipeline Upload test ##################### ################### Global Setup before each test ##################### Test Context: 0x268ac9835100 test-secret-1 created test-secret-2 created test-secret-3 created < Exit [BeforeEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:49 @ 07/03/26 11:55:34.945 (34ms) > Enter [It] Upload essential/iris_pipeline_compiled.yaml pipeline - /dspa/backend/test/end2end/pipeline_e2e_test.go:152 @ 07/03/26 11:55:34.945 Uploading pipeline file essential/iris_pipeline_compiled.yaml Creating temp pipeline file with overridden SDK Version Uploading pipeline with name=e2e-test-1783079734911491593, from file /dspa/test_data/sdk_compiled_pipelines/valid/essential/iris_pipeline_compiled.yaml Upload of pipeline file 'essential/iris_pipeline_compiled.yaml' successful Listing pipeline versions for pipeline eb7cd4f2-878b-4298-9c05-de3a49f6c494 Get the pipeline run time inputs from pipeline spec file /dspa/test_data/sdk_compiled_pipelines/valid/essential/iris_pipeline_compiled.yaml Returning pipeline run time inputs map[] Create run for pipeline with id: 'eb7cd4f2-878b-4298-9c05-de3a49f6c494' and name: 'e2e-test-1783079734911491593' Create a pipeline run for pipeline with id=eb7cd4f2-878b-4298-9c05-de3a49f6c494 and versionId=7cdbe344-f9f4-45f0-9df6-0ec149946a79 Create a pipeline run body Created Pipeline Run successfully with runId=cb986076-c4f2-4191-b98e-119e634220a4 Created Pipeline Run with id: cb986076-c4f2-4191-b98e-119e634220a4 for pipeline with id: eb7cd4f2-878b-4298-9c05-de3a49f6c494 Waiting for pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 to be in one of '[SUCCEEDED SKIPPED FAILED CANCELED]' Get a pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 Pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 is in PENDING state, waiting... Get a pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 Pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 is in RUNNING state, waiting... Get a pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 Pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 is in RUNNING state, waiting... Get a pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 Pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 is in RUNNING state, waiting... Get a pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 Pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 is in RUNNING state, waiting... Get a pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 Pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 is in RUNNING state, waiting... Get a pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 Pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 is in RUNNING state, waiting... Get a pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 Pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 is in RUNNING state, waiting... Get a pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 Pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 is in RUNNING state, waiting... Get a pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 Pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 is in RUNNING state, waiting... Get a pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 Pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 is in RUNNING state, waiting... Get a pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 Pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 is in RUNNING state, waiting... Get a pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 [FAILED] Failure recorded during attempt 1: Failed to get run with id=cb986076-c4f2-4191-b98e-119e634220a4 Unexpected error: <*util.UserError | 0x268aca33a1b0>: Failed to get run. Params: '&{ExperimentID:<nil> RunID:cb986076-c4f2-4191-b98e-119e634220a4 timeout:0 Context:context.Background.WithDeadline(2026-07-03 11:57:10.340227649 +0000 UTC m=+95.955017865 [25.276899074s]) HTTPClient:<nil>}': issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Get "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/runs/cb986076-c4f2-4191-b98e-119e634220a4": EOF { internalError: <*errors.withStack | 0x268aca134fd8>{ error: <*errors.withMessage | 0x268aca11ba40>{ cause: <*errors.errorString | 0x268aca3290b0>{ s: "issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Get \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/runs/cb986076-c4f2-4191-b98e-119e634220a4\": EOF", }, msg: "Failed to get run. Params: '&{ExperimentID:<nil> RunID:cb986076-c4f2-4191-b98e-119e634220a4 timeout:0 Context:context.Background.WithDeadline(2026-07-03 11:57:10.340227649 +0000 UTC m=+95.955017865 [25.276899074s]) HTTPClient:<nil>}'", }, stack: [0x1a01bf8, 0x2a124bd, 0x2b5f09b, 0x2b5f30c, 0x2b622ec, 0x2b74098, 0x2b72a2a, 0x2b1a153, 0x2b33a99, 0x49a241], }, externalMessage: "Failed to get run 'cb986076-c4f2-4191-b98e-119e634220a4'. Raw error from the service: issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Get \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/runs/cb986076-c4f2-4191-b98e-119e634220a4\": EOF", externalStatusCode: 13, } occurred In [It] at: /dspa/backend/test/testutil/pipeline_run_utils.go:82 @ 07/03/26 11:56:45.063 < Exit [It] Upload essential/iris_pipeline_compiled.yaml pipeline - /dspa/backend/test/end2end/pipeline_e2e_test.go:152 @ 07/03/26 11:56:45.063 (1m10.118s) > Enter [DeferCleanup (Each)] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:96 @ 07/03/26 11:56:45.063 ################### Global Cleanup after each test ##################### Deleting 1 run(s) Skipping Termination of the run cb986076-c4f2-4191-b98e-119e634220a4, because it does not exist Skipping Archiving of the run cb986076-c4f2-4191-b98e-119e634220a4, because it does not exist Skipping Deletion of the run cb986076-c4f2-4191-b98e-119e634220a4, as it does not exist Deleting 0 experiment(s) Deleting 1 pipeline(s) Deleting pipeline with id=eb7cd4f2-878b-4298-9c05-de3a49f6c494 (cascade=true) [FAILED] Error occurred while deleting pipeline with id=eb7cd4f2-878b-4298-9c05-de3a49f6c494 Unexpected error: <*util.UserError | 0x268aca385980>: Failed to delete pipeline. Params: '&{Cascade:0x268ac9ded02b PipelineID:eb7cd4f2-878b-4298-9c05-de3a49f6c494 timeout:0 Context:context.Background.WithDeadline(2026-07-03 11:57:20.081817979 +0000 UTC m=+105.696608194 [34.99405215s]) HTTPClient:<nil>}': issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Delete "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/pipelines/eb7cd4f2-878b-4298-9c05-de3a49f6c494?cascade=true": EOF { internalError: <*errors.withStack | 0x268ac96df458>{ error: <*errors.withMessage | 0x268aca0f78e0>{ cause: <*errors.errorString | 0x268aca461ce0>{ s: "issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Delete \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/pipelines/eb7cd4f2-878b-4298-9c05-de3a49f6c494?cascade=true\": EOF", }, msg: "Failed to delete pipeline. Params: '&{Cascade:0x268ac9ded02b PipelineID:eb7cd4f2-878b-4298-9c05-de3a49f6c494 timeout:0 Context:context.Background.WithDeadline(2026-07-03 11:57:20.081817979 +0000 UTC m=+105.696608194 [34.99405215s]) HTTPClient:<nil>}'", }, stack: [0x1a01bf8, 0x2a0d673, 0x2b60255, 0x2b74532, 0x2b742a9, 0x511506, 0x5103b9, 0x2b1bf54, 0x2b1a153, 0x2b33a99, 0x49a241], }, externalMessage: "Failed to delete pipeline 'eb7cd4f2-878b-4298-9c05-de3a49f6c494'. Raw error from the service: issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Delete \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/pipelines/eb7cd4f2-878b-4298-9c05-de3a49f6c494?cascade=true\": EOF", externalStatusCode: 13, } occurred In [DeferCleanup (Each)] at: /dspa/backend/test/end2end/pipeline_e2e_test.go:331 @ 07/03/26 11:56:45.088 < Exit [DeferCleanup (Each)] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:96 @ 07/03/26 11:56:45.088 (24ms) Attempt #1 Failed. Retrying ↺ @ 07/03/26 11:56:45.088 > Enter [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/03/26 11:56:45.088 Create an experiment with name E2EExperiment-1783079805088289777 [FAILED] Failed to create experiment Unexpected error: <*util.UserError | 0x268aca3ee3f0>: Failed to create experiment. Params: '&{Experiment:0x268ac9b7f580 timeout:30000000000 Context:context.Background.WithDeadline(2026-07-03 11:57:20.088292937 +0000 UTC m=+105.703083152 [34.994825836s]) HTTPClient:<nil>}'. Body: '&{CreatedAt:0001-01-01T00:00:00.000Z Description: DisplayName:E2EExperiment-1783079805088289777 ExperimentID: LastRunCreatedAt:0001-01-01T00:00:00.000Z Namespace:dspa-test-l5rgxfei StorageState:<nil>}': Post "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments": EOF { internalError: <*errors.withStack | 0x268ac96df908>{ error: <*errors.withMessage | 0x268aca0f7de0>{ cause: <*url.Error | 0x268aca3ee390>{ Op: "Post", URL: "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments", Err: <*errors.errorString | 0x5c15e50>{s: "EOF"}, }, msg: "Failed to create experiment. Params: '&{Experiment:0x268ac9b7f580 timeout:30000000000 Context:context.Background.WithDeadline(2026-07-03 11:57:20.088292937 +0000 UTC m=+105.703083152 [34.994825836s]) HTTPClient:<nil>}'. Body: '&{CreatedAt:0001-01-01T00:00:00.000Z Description: DisplayName:E2EExperiment-1783079805088289777 ExperimentID: LastRunCreatedAt:0001-01-01T00:00:00.000Z Namespace:dspa-test-l5rgxfei StorageState:<nil>}'", }, stack: [0x1a01bf8, 0x2a0c92c, 0x2b5bdcc, 0x2b6835c, 0x2b1a153, 0x2b33a99, 0x49a241], }, externalMessage: "Failed to create experiment 'E2EExperiment-1783079805088289777'. Raw error from the service: Post \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments\": EOF", externalStatusCode: 13, } occurred In [BeforeEach] at: /dspa/backend/test/testutil/experiment_utils.go:48 @ 07/03/26 11:56:45.093 < Exit [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/03/26 11:56:45.093 (5ms) > Enter [ReportAfterEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:99 @ 07/03/26 11:56:45.093 Argo Workflows archived logs - /dspa/backend/test/end2end/pipeline_e2e_test.go:105 @ 07/03/26 11:56:45.101 WARNING: failed to read mlpipeline-minio-artifact secret (dspa-test-l5rgxfei/mlpipeline-minio-artifact): secrets "mlpipeline-minio-artifact" not found < Exit [ReportAfterEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:99 @ 07/03/26 11:56:45.101 (7ms) > Enter [ReportAfterEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:155 @ 07/03/26 11:56:45.101 Test failed... Capturing logs Test Log - /dspa/backend/test/end2end/e2e_suite_test.go:158 @ 07/03/26 11:56:45.101 Create an experiment with name E2EExperiment-1783079734758596242 ################### Setup before each Pipeline Upload test ##################### ################### Global Setup before each test ##################### Test Context: 0x268ac9835100 test-secret-1 created test-secret-2 created test-secret-3 created Uploading pipeline file essential/iris_pipeline_compiled.yaml Creating temp pipeline file with overridden SDK Version Uploading pipeline with name=e2e-test-1783079734911491593, from file /dspa/test_data/sdk_compiled_pipelines/valid/essential/iris_pipeline_compiled.yaml Upload of pipeline file 'essential/iris_pipeline_compiled.yaml' successful Listing pipeline versions for pipeline eb7cd4f2-878b-4298-9c05-de3a49f6c494 Get the pipeline run time inputs from pipeline spec file /dspa/test_data/sdk_compiled_pipelines/valid/essential/iris_pipeline_compiled.yaml Returning pipeline run time inputs map[] Create run for pipeline with id: 'eb7cd4f2-878b-4298-9c05-de3a49f6c494' and name: 'e2e-test-1783079734911491593' Create a pipeline run for pipeline with id=eb7cd4f2-878b-4298-9c05-de3a49f6c494 and versionId=7cdbe344-f9f4-45f0-9df6-0ec149946a79 Create a pipeline run body Created Pipeline Run successfully with runId=cb986076-c4f2-4191-b98e-119e634220a4 Created Pipeline Run with id: cb986076-c4f2-4191-b98e-119e634220a4 for pipeline with id: eb7cd4f2-878b-4298-9c05-de3a49f6c494 Waiting for pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 to be in one of '[SUCCEEDED SKIPPED FAILED CANCELED]' Get a pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 Pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 is in PENDING state, waiting... Get a pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 Pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 is in RUNNING state, waiting... Get a pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 Pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 is in RUNNING state, waiting... Get a pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 Pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 is in RUNNING state, waiting... Get a pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 Pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 is in RUNNING state, waiting... Get a pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 Pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 is in RUNNING state, waiting... Get a pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 Pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 is in RUNNING state, waiting... Get a pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 Pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 is in RUNNING state, waiting... Get a pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 Pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 is in RUNNING state, waiting... Get a pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 Pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 is in RUNNING state, waiting... Get a pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 Pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 is in RUNNING state, waiting... Get a pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 Pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 is in RUNNING state, waiting... Get a pipeline run with id=cb986076-c4f2-4191-b98e-119e634220a4 ################### Global Cleanup after each test ##################### Deleting 1 run(s) Skipping Termination of the run cb986076-c4f2-4191-b98e-119e634220a4, because it does not exist Skipping Archiving of the run cb986076-c4f2-4191-b98e-119e634220a4, because it does not exist Skipping Deletion of the run cb986076-c4f2-4191-b98e-119e634220a4, as it does not exist Deleting 0 experiment(s) Deleting 1 pipeline(s) Deleting pipeline with id=eb7cd4f2-878b-4298-9c05-de3a49f6c494 (cascade=true) Create an experiment with name E2EExperiment-1783079805088289777 Failed to create log file due to: open /dspa/backend/test/end2end/logs/Upload and Verify Pipeline Run > Upload a pipeline file, run it and verify that pipeline run succeeds Smoke > Upload essential/iris_pipeline_compiled.yaml pipeline.log: no such file or directory Failed to write to the log file, due to: invalid argument < Exit [ReportAfterEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:155 @ 07/03/26 11:56:45.101 (0s) [FAILED] Failed to create experiment Unexpected error: <*util.UserError | 0x198fdedf32c0>: Failed to create experiment. Params: '&{Experiment:0x198fdedd9180 timeout:30000000000 Context:context.Background.WithDeadline(2026-07-03 11:57:20.094656642 +0000 UTC m=+105.709979525 [34.985845413s]) HTTPClient:<nil>}'. Body: '&{CreatedAt:0001-01-01T00:00:00.000Z Description: DisplayName:E2EExperiment-1783079805094652631 ExperimentID: LastRunCreatedAt:0001-01-01T00:00:00.000Z Namespace:dspa-test-l5rgxfei StorageState:<nil>}': Post "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments": EOF { internalError: <*errors.withStack | 0x198fdf34d188>{ error: <*errors.withMessage | 0x198fdeb95140>{ cause: <*url.Error | 0x198fdedf3260>{ Op: "Post", URL: "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments", Err: <*errors.errorString | 0x5c15e50>{s: "EOF"}, }, msg: "Failed to create experiment. Params: '&{Experiment:0x198fdedd9180 timeout:30000000000 Context:context.Background.WithDeadline(2026-07-03 11:57:20.094656642 +0000 UTC m=+105.709979525 [34.985845413s]) HTTPClient:<nil>}'. Body: '&{CreatedAt:0001-01-01T00:00:00.000Z Description: DisplayName:E2EExperiment-1783079805094652631 ExperimentID: LastRunCreatedAt:0001-01-01T00:00:00.000Z Namespace:dspa-test-l5rgxfei StorageState:<nil>}'", }, stack: [0x1a01bf8, 0x2a0c92c, 0x2b5bdcc, 0x2b6835c, 0x2b1a153, 0x2b33a99, 0x49a241], }, externalMessage: "Failed to create experiment 'E2EExperiment-1783079805094652631'. Raw error from the service: Post \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments\": EOF", externalStatusCode: 13, } occurred In [BeforeEach] at: /dspa/backend/test/testutil/experiment_utils.go:48 @ 07/03/26 11:56:45.108 There were additional failures detected after the initial failure. These are visible in the timeline > Enter [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/03/26 11:55:34.758 Create an experiment with name E2EExperiment-1783079734758591332 < Exit [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/03/26 11:55:34.911 (153ms) > Enter [BeforeEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:49 @ 07/03/26 11:55:34.911 ################### Setup before each Pipeline Upload test ##################### ################### Global Setup before each test ##################### Test Context: 0x198fde916500 Looks like test-secret-1 already exists, because creation failed due to secrets "test-secret-1" already exists Looks like test-secret-2 already exists, because creation failed due to secrets "test-secret-2" already exists Looks like test-secret-3 already exists, because creation failed due to secrets "test-secret-3" already exists < Exit [BeforeEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:49 @ 07/03/26 11:55:34.959 (48ms) > Enter [It] Upload essential/component_with_pip_index_urls.yaml pipeline - /dspa/backend/test/end2end/pipeline_e2e_test.go:152 @ 07/03/26 11:55:34.959 Uploading pipeline file essential/component_with_pip_index_urls.yaml Creating temp pipeline file with overridden SDK Version Uploading pipeline with name=e2e-test-1783079734911534464, from file /dspa/test_data/sdk_compiled_pipelines/valid/essential/component_with_pip_index_urls.yaml Upload of pipeline file 'essential/component_with_pip_index_urls.yaml' successful Listing pipeline versions for pipeline 0bb1ab22-ab98-407f-ac41-853753ec9a8c Get the pipeline run time inputs from pipeline spec file /dspa/test_data/sdk_compiled_pipelines/valid/essential/component_with_pip_index_urls.yaml Returning pipeline run time inputs map[] Create run for pipeline with id: '0bb1ab22-ab98-407f-ac41-853753ec9a8c' and name: 'e2e-test-1783079734911534464' Create a pipeline run for pipeline with id=0bb1ab22-ab98-407f-ac41-853753ec9a8c and versionId=e0f9124e-b231-4a59-9223-e2ebcee02c57 Create a pipeline run body Created Pipeline Run successfully with runId=065bc67f-db80-483c-b6d7-08228c1758bd Created Pipeline Run with id: 065bc67f-db80-483c-b6d7-08228c1758bd for pipeline with id: 0bb1ab22-ab98-407f-ac41-853753ec9a8c Waiting for pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd to be in one of '[SUCCEEDED SKIPPED FAILED CANCELED]' Get a pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd Pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd is in PENDING state, waiting... Get a pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd Pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd is in RUNNING state, waiting... Get a pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd Pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd is in RUNNING state, waiting... Get a pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd Pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd is in RUNNING state, waiting... Get a pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd Pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd is in RUNNING state, waiting... Get a pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd Pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd is in RUNNING state, waiting... Get a pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd Pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd is in RUNNING state, waiting... Get a pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd Pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd is in RUNNING state, waiting... Get a pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd Pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd is in RUNNING state, waiting... Get a pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd Pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd is in RUNNING state, waiting... Get a pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd Pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd is in RUNNING state, waiting... Get a pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd Pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd is in RUNNING state, waiting... Get a pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd [FAILED] Failure recorded during attempt 1: Failed to get run with id=065bc67f-db80-483c-b6d7-08228c1758bd Unexpected error: <*util.UserError | 0x198fdedf2750>: Failed to get run. Params: '&{ExperimentID:<nil> RunID:065bc67f-db80-483c-b6d7-08228c1758bd timeout:0 Context:context.Background.WithDeadline(2026-07-03 11:57:10.312179654 +0000 UTC m=+95.927502537 [26.679389214s]) HTTPClient:<nil>}': issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Get "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/runs/065bc67f-db80-483c-b6d7-08228c1758bd": EOF { internalError: <*errors.withStack | 0x198fdf34cf00>{ error: <*errors.withMessage | 0x198fdeb946e0>{ cause: <*errors.errorString | 0x198fdedf8120>{ s: "issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Get \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/runs/065bc67f-db80-483c-b6d7-08228c1758bd\": EOF", }, msg: "Failed to get run. Params: '&{ExperimentID:<nil> RunID:065bc67f-db80-483c-b6d7-08228c1758bd timeout:0 Context:context.Background.WithDeadline(2026-07-03 11:57:10.312179654 +0000 UTC m=+95.927502537 [26.679389214s]) HTTPClient:<nil>}'", }, stack: [0x1a01bf8, 0x2a124bd, 0x2b5f09b, 0x2b5f30c, 0x2b622ec, 0x2b74098, 0x2b72a2a, 0x2b1a153, 0x2b33a99, 0x49a241], }, externalMessage: "Failed to get run '065bc67f-db80-483c-b6d7-08228c1758bd'. Raw error from the service: issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Get \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/runs/065bc67f-db80-483c-b6d7-08228c1758bd\": EOF", externalStatusCode: 13, } occurred In [It] at: /dspa/backend/test/testutil/pipeline_run_utils.go:82 @ 07/03/26 11:56:43.633 < Exit [It] Upload essential/component_with_pip_index_urls.yaml pipeline - /dspa/backend/test/end2end/pipeline_e2e_test.go:152 @ 07/03/26 11:56:43.633 (1m8.673s) > Enter [DeferCleanup (Each)] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:96 @ 07/03/26 11:56:43.633 ################### Global Cleanup after each test ##################### Deleting 1 run(s) Skipping Termination of the run 065bc67f-db80-483c-b6d7-08228c1758bd, because it does not exist Skipping Archiving of the run 065bc67f-db80-483c-b6d7-08228c1758bd, because it does not exist Skipping Deletion of the run 065bc67f-db80-483c-b6d7-08228c1758bd, as it does not exist Deleting 0 experiment(s) Deleting 1 pipeline(s) Deleting pipeline with id=0bb1ab22-ab98-407f-ac41-853753ec9a8c (cascade=true) [FAILED] Error occurred while deleting pipeline with id=0bb1ab22-ab98-407f-ac41-853753ec9a8c Unexpected error: <*util.UserError | 0x198fdeff5a40>: Failed to delete pipeline. Params: '&{Cascade:0x198fde48f68b PipelineID:0bb1ab22-ab98-407f-ac41-853753ec9a8c timeout:0 Context:context.Background.WithDeadline(2026-07-03 11:57:20.088733652 +0000 UTC m=+105.704056535 [34.994633189s]) HTTPClient:<nil>}': issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Delete "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/pipelines/0bb1ab22-ab98-407f-ac41-853753ec9a8c?cascade=true": EOF { internalError: <*errors.withStack | 0x198fde4b94e8>{ error: <*errors.withMessage | 0x198fde430b20>{ cause: <*errors.errorString | 0x198fdefe90a0>{ s: "issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Delete \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/pipelines/0bb1ab22-ab98-407f-ac41-853753ec9a8c?cascade=true\": EOF", }, msg: "Failed to delete pipeline. Params: '&{Cascade:0x198fde48f68b PipelineID:0bb1ab22-ab98-407f-ac41-853753ec9a8c timeout:0 Context:context.Background.WithDeadline(2026-07-03 11:57:20.088733652 +0000 UTC m=+105.704056535 [34.994633189s]) HTTPClient:<nil>}'", }, stack: [0x1a01bf8, 0x2a0d673, 0x2b60255, 0x2b74532, 0x2b742a9, 0x511506, 0x5103b9, 0x2b1bf54, 0x2b1a153, 0x2b33a99, 0x49a241], }, externalMessage: "Failed to delete pipeline '0bb1ab22-ab98-407f-ac41-853753ec9a8c'. Raw error from the service: issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Delete \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/pipelines/0bb1ab22-ab98-407f-ac41-853753ec9a8c?cascade=true\": EOF", externalStatusCode: 13, } occurred In [DeferCleanup (Each)] at: /dspa/backend/test/end2end/pipeline_e2e_test.go:331 @ 07/03/26 11:56:45.094 < Exit [DeferCleanup (Each)] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:96 @ 07/03/26 11:56:45.094 (1.461s) Attempt #1 Failed. Retrying ↺ @ 07/03/26 11:56:45.094 > Enter [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/03/26 11:56:45.094 Create an experiment with name E2EExperiment-1783079805094652631 [FAILED] Failed to create experiment Unexpected error: <*util.UserError | 0x198fdedf32c0>: Failed to create experiment. Params: '&{Experiment:0x198fdedd9180 timeout:30000000000 Context:context.Background.WithDeadline(2026-07-03 11:57:20.094656642 +0000 UTC m=+105.709979525 [34.985845413s]) HTTPClient:<nil>}'. Body: '&{CreatedAt:0001-01-01T00:00:00.000Z Description: DisplayName:E2EExperiment-1783079805094652631 ExperimentID: LastRunCreatedAt:0001-01-01T00:00:00.000Z Namespace:dspa-test-l5rgxfei StorageState:<nil>}': Post "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments": EOF { internalError: <*errors.withStack | 0x198fdf34d188>{ error: <*errors.withMessage | 0x198fdeb95140>{ cause: <*url.Error | 0x198fdedf3260>{ Op: "Post", URL: "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments", Err: <*errors.errorString | 0x5c15e50>{s: "EOF"}, }, msg: "Failed to create experiment. Params: '&{Experiment:0x198fdedd9180 timeout:30000000000 Context:context.Background.WithDeadline(2026-07-03 11:57:20.094656642 +0000 UTC m=+105.709979525 [34.985845413s]) HTTPClient:<nil>}'. Body: '&{CreatedAt:0001-01-01T00:00:00.000Z Description: DisplayName:E2EExperiment-1783079805094652631 ExperimentID: LastRunCreatedAt:0001-01-01T00:00:00.000Z Namespace:dspa-test-l5rgxfei StorageState:<nil>}'", }, stack: [0x1a01bf8, 0x2a0c92c, 0x2b5bdcc, 0x2b6835c, 0x2b1a153, 0x2b33a99, 0x49a241], }, externalMessage: "Failed to create experiment 'E2EExperiment-1783079805094652631'. Raw error from the service: Post \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments\": EOF", externalStatusCode: 13, } occurred In [BeforeEach] at: /dspa/backend/test/testutil/experiment_utils.go:48 @ 07/03/26 11:56:45.108 < Exit [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/03/26 11:56:45.109 (14ms) > Enter [ReportAfterEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:99 @ 07/03/26 11:56:45.109 Argo Workflows archived logs - /dspa/backend/test/end2end/pipeline_e2e_test.go:105 @ 07/03/26 11:56:45.114 WARNING: failed to read mlpipeline-minio-artifact secret (dspa-test-l5rgxfei/mlpipeline-minio-artifact): secrets "mlpipeline-minio-artifact" not found < Exit [ReportAfterEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:99 @ 07/03/26 11:56:45.114 (5ms) > Enter [ReportAfterEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:155 @ 07/03/26 11:56:45.114 Test failed... Capturing logs Test Log - /dspa/backend/test/end2end/e2e_suite_test.go:158 @ 07/03/26 11:56:45.114 Create an experiment with name E2EExperiment-1783079734758591332 ################### Setup before each Pipeline Upload test ##################### ################### Global Setup before each test ##################### Test Context: 0x198fde916500 Looks like test-secret-1 already exists, because creation failed due to secrets "test-secret-1" already exists Looks like test-secret-2 already exists, because creation failed due to secrets "test-secret-2" already exists Looks like test-secret-3 already exists, because creation failed due to secrets "test-secret-3" already exists Uploading pipeline file essential/component_with_pip_index_urls.yaml Creating temp pipeline file with overridden SDK Version Uploading pipeline with name=e2e-test-1783079734911534464, from file /dspa/test_data/sdk_compiled_pipelines/valid/essential/component_with_pip_index_urls.yaml Upload of pipeline file 'essential/component_with_pip_index_urls.yaml' successful Listing pipeline versions for pipeline 0bb1ab22-ab98-407f-ac41-853753ec9a8c Get the pipeline run time inputs from pipeline spec file /dspa/test_data/sdk_compiled_pipelines/valid/essential/component_with_pip_index_urls.yaml Returning pipeline run time inputs map[] Create run for pipeline with id: '0bb1ab22-ab98-407f-ac41-853753ec9a8c' and name: 'e2e-test-1783079734911534464' Create a pipeline run for pipeline with id=0bb1ab22-ab98-407f-ac41-853753ec9a8c and versionId=e0f9124e-b231-4a59-9223-e2ebcee02c57 Create a pipeline run body Created Pipeline Run successfully with runId=065bc67f-db80-483c-b6d7-08228c1758bd Created Pipeline Run with id: 065bc67f-db80-483c-b6d7-08228c1758bd for pipeline with id: 0bb1ab22-ab98-407f-ac41-853753ec9a8c Waiting for pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd to be in one of '[SUCCEEDED SKIPPED FAILED CANCELED]' Get a pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd Pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd is in PENDING state, waiting... Get a pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd Pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd is in RUNNING state, waiting... Get a pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd Pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd is in RUNNING state, waiting... Get a pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd Pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd is in RUNNING state, waiting... Get a pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd Pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd is in RUNNING state, waiting... Get a pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd Pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd is in RUNNING state, waiting... Get a pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd Pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd is in RUNNING state, waiting... Get a pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd Pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd is in RUNNING state, waiting... Get a pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd Pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd is in RUNNING state, waiting... Get a pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd Pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd is in RUNNING state, waiting... Get a pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd Pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd is in RUNNING state, waiting... Get a pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd Pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd is in RUNNING state, waiting... Get a pipeline run with id=065bc67f-db80-483c-b6d7-08228c1758bd ################### Global Cleanup after each test ##################### Deleting 1 run(s) Skipping Termination of the run 065bc67f-db80-483c-b6d7-08228c1758bd, because it does not exist Skipping Archiving of the run 065bc67f-db80-483c-b6d7-08228c1758bd, because it does not exist Skipping Deletion of the run 065bc67f-db80-483c-b6d7-08228c1758bd, as it does not exist Deleting 0 experiment(s) Deleting 1 pipeline(s) Deleting pipeline with id=0bb1ab22-ab98-407f-ac41-853753ec9a8c (cascade=true) Create an experiment with name E2EExperiment-1783079805094652631 Failed to create log file due to: open /dspa/backend/test/end2end/logs/Upload and Verify Pipeline Run > Upload a pipeline file, run it and verify that pipeline run succeeds Smoke > Upload essential/component_with_pip_index_urls.yaml pipeline.log: no such file or directory Failed to write to the log file, due to: invalid argument < Exit [ReportAfterEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:155 @ 07/03/26 11:56:45.114 (0s) [FAILED] Failed to create experiment Unexpected error: <*util.UserError | 0x22eae75c8840>: Failed to create experiment. Params: '&{Experiment:0x22eae7291500 timeout:30000000000 Context:context.Background.WithDeadline(2026-07-03 11:57:20.070785607 +0000 UTC m=+105.688278362 [34.993608094s]) HTTPClient:<nil>}'. Body: '&{CreatedAt:0001-01-01T00:00:00.000Z Description: DisplayName:E2EExperiment-1783079805070782037 ExperimentID: LastRunCreatedAt:0001-01-01T00:00:00.000Z Namespace:dspa-test-l5rgxfei StorageState:<nil>}': Post "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments": EOF { internalError: <*errors.withStack | 0x22eae78ff518>{ error: <*errors.withMessage | 0x22eae75b2160>{ cause: <*url.Error | 0x22eae75c87e0>{ Op: "Post", URL: "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments", Err: <*errors.errorString | 0x5c15e50>{s: "EOF"}, }, msg: "Failed to create experiment. Params: '&{Experiment:0x22eae7291500 timeout:30000000000 Context:context.Background.WithDeadline(2026-07-03 11:57:20.070785607 +0000 UTC m=+105.688278362 [34.993608094s]) HTTPClient:<nil>}'. Body: '&{CreatedAt:0001-01-01T00:00:00.000Z Description: DisplayName:E2EExperiment-1783079805070782037 ExperimentID: LastRunCreatedAt:0001-01-01T00:00:00.000Z Namespace:dspa-test-l5rgxfei StorageState:<nil>}'", }, stack: [0x1a01bf8, 0x2a0c92c, 0x2b5bdcc, 0x2b6835c, 0x2b1a153, 0x2b33a99, 0x49a241], }, externalMessage: "Failed to create experiment 'E2EExperiment-1783079805070782037'. Raw error from the service: Post \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments\": EOF", externalStatusCode: 13, } occurred In [BeforeEach] at: /dspa/backend/test/testutil/experiment_utils.go:48 @ 07/03/26 11:56:45.077 There were additional failures detected after the initial failure. These are visible in the timeline > Enter [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/03/26 11:55:34.758 Create an experiment with name E2EExperiment-1783079734758067634 < Exit [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/03/26 11:55:34.914 (157ms) > Enter [BeforeEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:49 @ 07/03/26 11:55:34.914 ################### Setup before each Pipeline Upload test ##################### ################### Global Setup before each test ##################### Test Context: 0x22eae74fe200 Looks like test-secret-1 already exists, because creation failed due to secrets "test-secret-1" already exists Looks like test-secret-2 already exists, because creation failed due to secrets "test-secret-2" already exists Looks like test-secret-3 already exists, because creation failed due to secrets "test-secret-3" already exists < Exit [BeforeEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:49 @ 07/03/26 11:55:34.964 (50ms) > Enter [It] Upload critical/pipeline_with_artifact_upload_download.yaml pipeline - /dspa/backend/test/end2end/pipeline_e2e_test.go:152 @ 07/03/26 11:55:34.964 Uploading pipeline file critical/pipeline_with_artifact_upload_download.yaml Creating temp pipeline file with overridden SDK Version Uploading pipeline with name=e2e-test-1783079734914725351, from file /dspa/test_data/sdk_compiled_pipelines/valid/critical/pipeline_with_artifact_upload_download.yaml Upload of pipeline file 'critical/pipeline_with_artifact_upload_download.yaml' successful Listing pipeline versions for pipeline 6fa66086-9645-44b3-ade6-9ab95df44a9e Get the pipeline run time inputs from pipeline spec file /dspa/test_data/sdk_compiled_pipelines/valid/critical/pipeline_with_artifact_upload_download.yaml Returning pipeline run time inputs map[] Create run for pipeline with id: '6fa66086-9645-44b3-ade6-9ab95df44a9e' and name: 'e2e-test-1783079734914725351' Create a pipeline run for pipeline with id=6fa66086-9645-44b3-ade6-9ab95df44a9e and versionId=deac6bac-7d4e-4355-b66e-0d9b2cbf081c Create a pipeline run body Created Pipeline Run successfully with runId=5ee3393f-9441-4351-a414-5a39c3244743 Created Pipeline Run with id: 5ee3393f-9441-4351-a414-5a39c3244743 for pipeline with id: 6fa66086-9645-44b3-ade6-9ab95df44a9e Waiting for pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 to be in one of '[SUCCEEDED SKIPPED FAILED CANCELED]' Get a pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 Pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 is in PENDING state, waiting... Get a pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 Pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 is in RUNNING state, waiting... Get a pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 Pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 is in RUNNING state, waiting... Get a pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 Pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 is in RUNNING state, waiting... Get a pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 Pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 is in RUNNING state, waiting... Get a pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 Pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 is in RUNNING state, waiting... Get a pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 Pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 is in RUNNING state, waiting... Get a pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 Pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 is in RUNNING state, waiting... Get a pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 Pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 is in RUNNING state, waiting... Get a pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 Pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 is in RUNNING state, waiting... Get a pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 Pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 is in RUNNING state, waiting... Get a pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 Pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 is in RUNNING state, waiting... Get a pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 [FAILED] Failure recorded during attempt 1: Failed to get run with id=5ee3393f-9441-4351-a414-5a39c3244743 Unexpected error: <*util.UserError | 0x22eae7b7dd10>: Failed to get run. Params: '&{ExperimentID:<nil> RunID:5ee3393f-9441-4351-a414-5a39c3244743 timeout:0 Context:context.Background.WithDeadline(2026-07-03 11:57:10.352740381 +0000 UTC m=+95.970233146 [26.719974322s]) HTTPClient:<nil>}': issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Get "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/runs/5ee3393f-9441-4351-a414-5a39c3244743": EOF { internalError: <*errors.withStack | 0x22eae7512588>{ error: <*errors.withMessage | 0x22eae7b2a940>{ cause: <*errors.errorString | 0x22eae74b9ba0>{ s: "issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Get \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/runs/5ee3393f-9441-4351-a414-5a39c3244743\": EOF", }, msg: "Failed to get run. Params: '&{ExperimentID:<nil> RunID:5ee3393f-9441-4351-a414-5a39c3244743 timeout:0 Context:context.Background.WithDeadline(2026-07-03 11:57:10.352740381 +0000 UTC m=+95.970233146 [26.719974322s]) HTTPClient:<nil>}'", }, stack: [0x1a01bf8, 0x2a124bd, 0x2b5f09b, 0x2b5f30c, 0x2b622ec, 0x2b74098, 0x2b72a2a, 0x2b1a153, 0x2b33a99, 0x49a241], }, externalMessage: "Failed to get run '5ee3393f-9441-4351-a414-5a39c3244743'. Raw error from the service: issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Get \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/runs/5ee3393f-9441-4351-a414-5a39c3244743\": EOF", externalStatusCode: 13, } occurred In [It] at: /dspa/backend/test/testutil/pipeline_run_utils.go:82 @ 07/03/26 11:56:43.633 < Exit [It] Upload critical/pipeline_with_artifact_upload_download.yaml pipeline - /dspa/backend/test/end2end/pipeline_e2e_test.go:152 @ 07/03/26 11:56:43.633 (1m8.669s) > Enter [DeferCleanup (Each)] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:96 @ 07/03/26 11:56:43.633 ################### Global Cleanup after each test ##################### Deleting 1 run(s) Skipping Termination of the run 5ee3393f-9441-4351-a414-5a39c3244743, because it does not exist Skipping Archiving of the run 5ee3393f-9441-4351-a414-5a39c3244743, because it does not exist Skipping Deletion of the run 5ee3393f-9441-4351-a414-5a39c3244743, as it does not exist Deleting 0 experiment(s) Deleting 1 pipeline(s) Deleting pipeline with id=6fa66086-9645-44b3-ade6-9ab95df44a9e (cascade=true) [FAILED] Error occurred while deleting pipeline with id=6fa66086-9645-44b3-ade6-9ab95df44a9e Unexpected error: <*util.UserError | 0x22eae7415e60>: Failed to delete pipeline. Params: '&{Cascade:0x22eae71f97db PipelineID:6fa66086-9645-44b3-ade6-9ab95df44a9e timeout:0 Context:context.Background.WithDeadline(2026-07-03 11:57:20.063333596 +0000 UTC m=+105.680826361 [34.993268653s]) HTTPClient:<nil>}': issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Delete "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/pipelines/6fa66086-9645-44b3-ade6-9ab95df44a9e?cascade=true": EOF { internalError: <*errors.withStack | 0x22eae78ff308>{ error: <*errors.withMessage | 0x22eae72dbde0>{ cause: <*errors.errorString | 0x22eae74119d0>{ s: "issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Delete \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/pipelines/6fa66086-9645-44b3-ade6-9ab95df44a9e?cascade=true\": EOF", }, msg: "Failed to delete pipeline. Params: '&{Cascade:0x22eae71f97db PipelineID:6fa66086-9645-44b3-ade6-9ab95df44a9e timeout:0 Context:context.Background.WithDeadline(2026-07-03 11:57:20.063333596 +0000 UTC m=+105.680826361 [34.993268653s]) HTTPClient:<nil>}'", }, stack: [0x1a01bf8, 0x2a0d673, 0x2b60255, 0x2b74532, 0x2b742a9, 0x511506, 0x5103b9, 0x2b1bf54, 0x2b1a153, 0x2b33a99, 0x49a241], }, externalMessage: "Failed to delete pipeline '6fa66086-9645-44b3-ade6-9ab95df44a9e'. Raw error from the service: issue calling the service. Use the '--debug' flag to see the HTTP request/response. Raw error from the client: Delete \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/pipelines/6fa66086-9645-44b3-ade6-9ab95df44a9e?cascade=true\": EOF", externalStatusCode: 13, } occurred In [DeferCleanup (Each)] at: /dspa/backend/test/end2end/pipeline_e2e_test.go:331 @ 07/03/26 11:56:45.07 < Exit [DeferCleanup (Each)] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:96 @ 07/03/26 11:56:45.07 (1.437s) Attempt #1 Failed. Retrying ↺ @ 07/03/26 11:56:45.07 > Enter [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/03/26 11:56:45.07 Create an experiment with name E2EExperiment-1783079805070782037 [FAILED] Failed to create experiment Unexpected error: <*util.UserError | 0x22eae75c8840>: Failed to create experiment. Params: '&{Experiment:0x22eae7291500 timeout:30000000000 Context:context.Background.WithDeadline(2026-07-03 11:57:20.070785607 +0000 UTC m=+105.688278362 [34.993608094s]) HTTPClient:<nil>}'. Body: '&{CreatedAt:0001-01-01T00:00:00.000Z Description: DisplayName:E2EExperiment-1783079805070782037 ExperimentID: LastRunCreatedAt:0001-01-01T00:00:00.000Z Namespace:dspa-test-l5rgxfei StorageState:<nil>}': Post "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments": EOF { internalError: <*errors.withStack | 0x22eae78ff518>{ error: <*errors.withMessage | 0x22eae75b2160>{ cause: <*url.Error | 0x22eae75c87e0>{ Op: "Post", URL: "https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments", Err: <*errors.errorString | 0x5c15e50>{s: "EOF"}, }, msg: "Failed to create experiment. Params: '&{Experiment:0x22eae7291500 timeout:30000000000 Context:context.Background.WithDeadline(2026-07-03 11:57:20.070785607 +0000 UTC m=+105.688278362 [34.993608094s]) HTTPClient:<nil>}'. Body: '&{CreatedAt:0001-01-01T00:00:00.000Z Description: DisplayName:E2EExperiment-1783079805070782037 ExperimentID: LastRunCreatedAt:0001-01-01T00:00:00.000Z Namespace:dspa-test-l5rgxfei StorageState:<nil>}'", }, stack: [0x1a01bf8, 0x2a0c92c, 0x2b5bdcc, 0x2b6835c, 0x2b1a153, 0x2b33a99, 0x49a241], }, externalMessage: "Failed to create experiment 'E2EExperiment-1783079805070782037'. Raw error from the service: Post \"https://ds-pipeline-dspa-dspa-test-l5rgxfei.apps.8c86ac5e-f85e-4a7f-87c4-154974f5f0e3.prod.konfluxeaas.com/apis/v2beta1/experiments\": EOF", externalStatusCode: 13, } occurred In [BeforeEach] at: /dspa/backend/test/testutil/experiment_utils.go:48 @ 07/03/26 11:56:45.077 < Exit [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/03/26 11:56:45.077 (7ms) > Enter [ReportAfterEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:99 @ 07/03/26 11:56:45.077 Argo Workflows archived logs - /dspa/backend/test/end2end/pipeline_e2e_test.go:105 @ 07/03/26 11:56:45.126 WARNING: failed to read mlpipeline-minio-artifact secret (dspa-test-l5rgxfei/mlpipeline-minio-artifact): secrets "mlpipeline-minio-artifact" not found < Exit [ReportAfterEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:99 @ 07/03/26 11:56:45.126 (49ms) > Enter [ReportAfterEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:155 @ 07/03/26 11:56:45.126 Test failed... Capturing logs Test Log - /dspa/backend/test/end2end/e2e_suite_test.go:158 @ 07/03/26 11:56:45.126 Create an experiment with name E2EExperiment-1783079734758067634 ################### Setup before each Pipeline Upload test ##################### ################### Global Setup before each test ##################### Test Context: 0x22eae74fe200 Looks like test-secret-1 already exists, because creation failed due to secrets "test-secret-1" already exists Looks like test-secret-2 already exists, because creation failed due to secrets "test-secret-2" already exists Looks like test-secret-3 already exists, because creation failed due to secrets "test-secret-3" already exists Uploading pipeline file critical/pipeline_with_artifact_upload_download.yaml Creating temp pipeline file with overridden SDK Version Uploading pipeline with name=e2e-test-1783079734914725351, from file /dspa/test_data/sdk_compiled_pipelines/valid/critical/pipeline_with_artifact_upload_download.yaml Upload of pipeline file 'critical/pipeline_with_artifact_upload_download.yaml' successful Listing pipeline versions for pipeline 6fa66086-9645-44b3-ade6-9ab95df44a9e Get the pipeline run time inputs from pipeline spec file /dspa/test_data/sdk_compiled_pipelines/valid/critical/pipeline_with_artifact_upload_download.yaml Returning pipeline run time inputs map[] Create run for pipeline with id: '6fa66086-9645-44b3-ade6-9ab95df44a9e' and name: 'e2e-test-1783079734914725351' Create a pipeline run for pipeline with id=6fa66086-9645-44b3-ade6-9ab95df44a9e and versionId=deac6bac-7d4e-4355-b66e-0d9b2cbf081c Create a pipeline run body Created Pipeline Run successfully with runId=5ee3393f-9441-4351-a414-5a39c3244743 Created Pipeline Run with id: 5ee3393f-9441-4351-a414-5a39c3244743 for pipeline with id: 6fa66086-9645-44b3-ade6-9ab95df44a9e Waiting for pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 to be in one of '[SUCCEEDED SKIPPED FAILED CANCELED]' Get a pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 Pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 is in PENDING state, waiting... Get a pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 Pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 is in RUNNING state, waiting... Get a pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 Pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 is in RUNNING state, waiting... Get a pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 Pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 is in RUNNING state, waiting... Get a pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 Pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 is in RUNNING state, waiting... Get a pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 Pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 is in RUNNING state, waiting... Get a pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 Pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 is in RUNNING state, waiting... Get a pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 Pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 is in RUNNING state, waiting... Get a pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 Pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 is in RUNNING state, waiting... Get a pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 Pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 is in RUNNING state, waiting... Get a pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 Pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 is in RUNNING state, waiting... Get a pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 Pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 is in RUNNING state, waiting... Get a pipeline run with id=5ee3393f-9441-4351-a414-5a39c3244743 ################### Global Cleanup after each test ##################### Deleting 1 run(s) Skipping Termination of the run 5ee3393f-9441-4351-a414-5a39c3244743, because it does not exist Skipping Archiving of the run 5ee3393f-9441-4351-a414-5a39c3244743, because it does not exist Skipping Deletion of the run 5ee3393f-9441-4351-a414-5a39c3244743, as it does not exist Deleting 0 experiment(s) Deleting 1 pipeline(s) Deleting pipeline with id=6fa66086-9645-44b3-ade6-9ab95df44a9e (cascade=true) Create an experiment with name E2EExperiment-1783079805070782037 Failed to create log file due to: open /dspa/backend/test/end2end/logs/Upload and Verify Pipeline Run > Upload a pipeline file, run it and verify that pipeline run succeeds Smoke > Upload critical/pipeline_with_artifact_upload_download.yaml pipeline.log: no such file or directory Failed to write to the log file, due to: invalid argument < Exit [ReportAfterEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:155 @ 07/03/26 11:56:45.126 (0s)