2026/07/15 04:35:14 Test passed > Enter [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/15/26 04:34:28.591 Create an experiment with name E2EExperiment-1784090068591300429 < Exit [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/15/26 04:34:28.802 (211ms) > Enter [BeforeEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:49 @ 07/15/26 04:34:28.802 ################### Setup before each Pipeline Upload test ##################### ################### Global Setup before each test ##################### Test Context: 0x38beb2a58300 Looks like test-secret-1 already exists, because creation failed due to secrets "test-secret-1" already exists Looks like test-secret-2 already exists, because creation failed due to secrets "test-secret-2" already exists Looks like test-secret-3 already exists, because creation failed due to secrets "test-secret-3" already exists < Exit [BeforeEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:49 @ 07/15/26 04:34:28.96 (158ms) > Enter [It] Upload essential/component_with_pip_index_urls.yaml pipeline - /dspa/backend/test/end2end/pipeline_e2e_test.go:152 @ 07/15/26 04:34:28.961 Uploading pipeline file essential/component_with_pip_index_urls.yaml Creating temp pipeline file with overridden SDK Version Uploading pipeline with name=e2e-test-1784090068802721181, from file /dspa/test_data/sdk_compiled_pipelines/valid/essential/component_with_pip_index_urls.yaml Upload of pipeline file 'essential/component_with_pip_index_urls.yaml' successful Listing pipeline versions for pipeline 31a2bad2-f1b7-4c51-943d-b1aec59ce308 Get the pipeline run time inputs from pipeline spec file /dspa/test_data/sdk_compiled_pipelines/valid/essential/component_with_pip_index_urls.yaml Returning pipeline run time inputs map[] Create run for pipeline with id: '31a2bad2-f1b7-4c51-943d-b1aec59ce308' and name: 'e2e-test-1784090068802721181' Create a pipeline run for pipeline with id=31a2bad2-f1b7-4c51-943d-b1aec59ce308 and versionId=60d709fc-9883-4435-af24-4f4a91dde248 Create a pipeline run body Created Pipeline Run successfully with runId=a86c7049-22df-43bc-bb2d-eaf9d9dea33a Created Pipeline Run with id: a86c7049-22df-43bc-bb2d-eaf9d9dea33a for pipeline with id: 31a2bad2-f1b7-4c51-943d-b1aec59ce308 Waiting for pipeline run with id=a86c7049-22df-43bc-bb2d-eaf9d9dea33a to be in one of '[SUCCEEDED SKIPPED FAILED CANCELED]' Get a pipeline run with id=a86c7049-22df-43bc-bb2d-eaf9d9dea33a Pipeline run with id=a86c7049-22df-43bc-bb2d-eaf9d9dea33a is in PENDING state, waiting... Get a pipeline run with id=a86c7049-22df-43bc-bb2d-eaf9d9dea33a Pipeline run with id=a86c7049-22df-43bc-bb2d-eaf9d9dea33a is in RUNNING state, waiting... Get a pipeline run with id=a86c7049-22df-43bc-bb2d-eaf9d9dea33a Pipeline run with id=a86c7049-22df-43bc-bb2d-eaf9d9dea33a is in RUNNING state, waiting... Get a pipeline run with id=a86c7049-22df-43bc-bb2d-eaf9d9dea33a Pipeline run with id=a86c7049-22df-43bc-bb2d-eaf9d9dea33a is in RUNNING state, waiting... Get a pipeline run with id=a86c7049-22df-43bc-bb2d-eaf9d9dea33a Pipeline run with id=a86c7049-22df-43bc-bb2d-eaf9d9dea33a is in RUNNING state, waiting... Get a pipeline run with id=a86c7049-22df-43bc-bb2d-eaf9d9dea33a Pipeline run with id=a86c7049-22df-43bc-bb2d-eaf9d9dea33a is in RUNNING state, waiting... Get a pipeline run with id=a86c7049-22df-43bc-bb2d-eaf9d9dea33a Pipeline run with id=a86c7049-22df-43bc-bb2d-eaf9d9dea33a is in RUNNING state, waiting... Get a pipeline run with id=a86c7049-22df-43bc-bb2d-eaf9d9dea33a Pipeline run with id=a86c7049-22df-43bc-bb2d-eaf9d9dea33a is in RUNNING state, waiting... Get a pipeline run with id=a86c7049-22df-43bc-bb2d-eaf9d9dea33a Pipeline run with id=a86c7049-22df-43bc-bb2d-eaf9d9dea33a is in RUNNING state, waiting... Get a pipeline run with id=a86c7049-22df-43bc-bb2d-eaf9d9dea33a Pipeline run with id=a86c7049-22df-43bc-bb2d-eaf9d9dea33a reached expected state SUCCEEDED Deserializing expected compiled workflow file 'essential/component_with_pip_index_urls.yaml' for the pipeline Unmarshalling Expected Workflow YAML Unmarshalled Expected Workflow YAML Fetching updated pipeline run details for run with id=a86c7049-22df-43bc-bb2d-eaf9d9dea33a Get a pipeline run with id=a86c7049-22df-43bc-bb2d-eaf9d9dea33a Updated pipeline run details Pipeline run succeeded, checking if the number of tasks are what is expected < Exit [It] Upload essential/component_with_pip_index_urls.yaml pipeline - /dspa/backend/test/end2end/pipeline_e2e_test.go:152 @ 07/15/26 04:35:14.417 (45.457s) > Enter [DeferCleanup (Each)] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:96 @ 07/15/26 04:35:14.417 ################### Global Cleanup after each test ##################### Deleting 1 run(s) Terminate run a86c7049-22df-43bc-bb2d-eaf9d9dea33a Failed to terminate run a86c7049-22df-43bc-bb2d-eaf9d9dea33a Terminate run a86c7049-22df-43bc-bb2d-eaf9d9dea33a Deleting run a86c7049-22df-43bc-bb2d-eaf9d9dea33a Deleting 0 experiment(s) Deleting 1 pipeline(s) Deleting pipeline with id=31a2bad2-f1b7-4c51-943d-b1aec59ce308 (cascade=true) Pipeline with id=31a2bad2-f1b7-4c51-943d-b1aec59ce308, DELETED < Exit [DeferCleanup (Each)] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:96 @ 07/15/26 04:35:14.875 (458ms) > Enter [ReportAfterEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:99 @ 07/15/26 04:35:14.876 < Exit [ReportAfterEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:99 @ 07/15/26 04:35:14.876 (0s) > Enter [ReportAfterEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:155 @ 07/15/26 04:35:14.876 < Exit [ReportAfterEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:155 @ 07/15/26 04:35:14.876 (0s) 2026/07/15 04:36:04 Test passed > Enter [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/15/26 04:34:28.59 Create an experiment with name E2EExperiment-1784090068590950631 < Exit [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/15/26 04:34:28.802 (212ms) > Enter [BeforeEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:49 @ 07/15/26 04:34:28.803 ################### Setup before each Pipeline Upload test ##################### ################### Global Setup before each test ##################### Test Context: 0xa367f53e580 Looks like test-secret-1 already exists, because creation failed due to secrets "test-secret-1" already exists Looks like test-secret-2 already exists, because creation failed due to secrets "test-secret-2" already exists Looks like test-secret-3 already exists, because creation failed due to secrets "test-secret-3" already exists < Exit [BeforeEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:49 @ 07/15/26 04:34:28.948 (145ms) > Enter [It] Upload critical/parallel_for_after_dependency.yaml pipeline - /dspa/backend/test/end2end/pipeline_e2e_test.go:152 @ 07/15/26 04:34:28.948 Uploading pipeline file critical/parallel_for_after_dependency.yaml Creating temp pipeline file with overridden SDK Version Uploading pipeline with name=e2e-test-1784090068803186782, from file /dspa/test_data/sdk_compiled_pipelines/valid/critical/parallel_for_after_dependency.yaml Upload of pipeline file 'critical/parallel_for_after_dependency.yaml' successful Listing pipeline versions for pipeline e8fff24d-9ed5-46c0-aca9-dc40b74a27cd Get the pipeline run time inputs from pipeline spec file /dspa/test_data/sdk_compiled_pipelines/valid/critical/parallel_for_after_dependency.yaml Returning pipeline run time inputs map[] Create run for pipeline with id: 'e8fff24d-9ed5-46c0-aca9-dc40b74a27cd' and name: 'e2e-test-1784090068803186782' Create a pipeline run for pipeline with id=e8fff24d-9ed5-46c0-aca9-dc40b74a27cd and versionId=0b9e8985-bbe2-4394-8f26-2912909df1bc Create a pipeline run body Created Pipeline Run successfully with runId=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Created Pipeline Run with id: 31bb77cd-38d5-46ea-8ce9-4ab3113126f3 for pipeline with id: e8fff24d-9ed5-46c0-aca9-dc40b74a27cd Waiting for pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 to be in one of '[SUCCEEDED SKIPPED FAILED CANCELED]' Get a pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 is in PENDING state, waiting... Get a pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 is in RUNNING state, waiting... Get a pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 is in RUNNING state, waiting... Get a pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 is in RUNNING state, waiting... Get a pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 is in RUNNING state, waiting... Get a pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 is in RUNNING state, waiting... Get a pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 is in RUNNING state, waiting... Get a pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 is in RUNNING state, waiting... Get a pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 is in RUNNING state, waiting... Get a pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 is in RUNNING state, waiting... Get a pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 is in RUNNING state, waiting... Get a pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 is in RUNNING state, waiting... Get a pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 is in RUNNING state, waiting... Get a pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 is in RUNNING state, waiting... Get a pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 is in RUNNING state, waiting... Get a pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 is in RUNNING state, waiting... Get a pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 is in RUNNING state, waiting... Get a pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 is in RUNNING state, waiting... Get a pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 is in RUNNING state, waiting... Get a pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 reached expected state SUCCEEDED Deserializing expected compiled workflow file 'critical/parallel_for_after_dependency.yaml' for the pipeline Unmarshalling Expected Workflow YAML Unmarshalled Expected Workflow YAML Fetching updated pipeline run details for run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Get a pipeline run with id=31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Updated pipeline run details Pipeline run succeeded, checking if the number of tasks are what is expected < Exit [It] Upload critical/parallel_for_after_dependency.yaml pipeline - /dspa/backend/test/end2end/pipeline_e2e_test.go:152 @ 07/15/26 04:36:04.428 (1m35.48s) > Enter [DeferCleanup (Each)] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:96 @ 07/15/26 04:36:04.428 ################### Global Cleanup after each test ##################### Deleting 1 run(s) Terminate run 31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Failed to terminate run 31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Terminate run 31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Deleting run 31bb77cd-38d5-46ea-8ce9-4ab3113126f3 Deleting 0 experiment(s) Deleting 1 pipeline(s) Deleting pipeline with id=e8fff24d-9ed5-46c0-aca9-dc40b74a27cd (cascade=true) Pipeline with id=e8fff24d-9ed5-46c0-aca9-dc40b74a27cd, DELETED < Exit [DeferCleanup (Each)] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:96 @ 07/15/26 04:36:04.818 (390ms) > Enter [ReportAfterEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:99 @ 07/15/26 04:36:04.818 < Exit [ReportAfterEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:99 @ 07/15/26 04:36:04.818 (0s) > Enter [ReportAfterEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:155 @ 07/15/26 04:36:04.818 < Exit [ReportAfterEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:155 @ 07/15/26 04:36:04.818 (0s) 2026/07/15 04:36:34 Test passed > Enter [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/15/26 04:34:28.59 Create an experiment with name E2EExperiment-1784090068590794457 < Exit [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/15/26 04:34:28.803 (212ms) > Enter [BeforeEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:49 @ 07/15/26 04:34:28.803 ################### Setup before each Pipeline Upload test ##################### ################### Global Setup before each test ##################### Test Context: 0x19832f324380 Looks like test-secret-1 already exists, because creation failed due to secrets "test-secret-1" already exists Looks like test-secret-2 already exists, because creation failed due to secrets "test-secret-2" already exists Looks like test-secret-3 already exists, because creation failed due to secrets "test-secret-3" already exists < Exit [BeforeEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:49 @ 07/15/26 04:34:28.944 (141ms) > Enter [It] Upload critical/flip_coin.yaml pipeline - /dspa/backend/test/end2end/pipeline_e2e_test.go:152 @ 07/15/26 04:34:28.944 Uploading pipeline file critical/flip_coin.yaml Creating temp pipeline file with overridden SDK Version Uploading pipeline with name=e2e-test-1784090068803238863, from file /dspa/test_data/sdk_compiled_pipelines/valid/critical/flip_coin.yaml Upload of pipeline file 'critical/flip_coin.yaml' successful Listing pipeline versions for pipeline 3777f4ad-b158-444f-8406-25d8f70e9ec3 Get the pipeline run time inputs from pipeline spec file /dspa/test_data/sdk_compiled_pipelines/valid/critical/flip_coin.yaml Returning pipeline run time inputs map[] Create run for pipeline with id: '3777f4ad-b158-444f-8406-25d8f70e9ec3' and name: 'e2e-test-1784090068803238863' Create a pipeline run for pipeline with id=3777f4ad-b158-444f-8406-25d8f70e9ec3 and versionId=6152ed1c-ecb3-46e1-85eb-0824a4687298 Create a pipeline run body Created Pipeline Run successfully with runId=f513e84a-51af-4de6-97d4-803518bb80e5 Created Pipeline Run with id: f513e84a-51af-4de6-97d4-803518bb80e5 for pipeline with id: 3777f4ad-b158-444f-8406-25d8f70e9ec3 Waiting for pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 to be in one of '[SUCCEEDED SKIPPED FAILED CANCELED]' Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 is in PENDING state, waiting... Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 is in RUNNING state, waiting... Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 is in RUNNING state, waiting... Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 is in RUNNING state, waiting... Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 is in RUNNING state, waiting... Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 is in RUNNING state, waiting... Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 is in RUNNING state, waiting... Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 is in RUNNING state, waiting... Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 is in RUNNING state, waiting... Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 is in RUNNING state, waiting... Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 is in RUNNING state, waiting... Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 is in RUNNING state, waiting... Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 is in RUNNING state, waiting... Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 is in RUNNING state, waiting... Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 is in RUNNING state, waiting... Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 is in RUNNING state, waiting... Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 is in RUNNING state, waiting... Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 is in RUNNING state, waiting... Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 is in RUNNING state, waiting... Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 is in RUNNING state, waiting... Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 is in RUNNING state, waiting... Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 is in RUNNING state, waiting... Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 is in RUNNING state, waiting... Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 is in RUNNING state, waiting... Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 is in RUNNING state, waiting... Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 reached expected state SUCCEEDED Deserializing expected compiled workflow file 'critical/flip_coin.yaml' for the pipeline Unmarshalling Expected Workflow YAML Unmarshalled Expected Workflow YAML Fetching updated pipeline run details for run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Get a pipeline run with id=f513e84a-51af-4de6-97d4-803518bb80e5 Updated pipeline run details Pipeline run succeeded, checking if the number of tasks are what is expected < Exit [It] Upload critical/flip_coin.yaml pipeline - /dspa/backend/test/end2end/pipeline_e2e_test.go:152 @ 07/15/26 04:36:34.775 (2m5.831s) > Enter [DeferCleanup (Each)] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:96 @ 07/15/26 04:36:34.775 ################### Global Cleanup after each test ##################### Deleting 1 run(s) Terminate run f513e84a-51af-4de6-97d4-803518bb80e5 Failed to terminate run f513e84a-51af-4de6-97d4-803518bb80e5 Terminate run f513e84a-51af-4de6-97d4-803518bb80e5 Deleting run f513e84a-51af-4de6-97d4-803518bb80e5 Deleting 0 experiment(s) Deleting 1 pipeline(s) Deleting pipeline with id=3777f4ad-b158-444f-8406-25d8f70e9ec3 (cascade=true) Pipeline with id=3777f4ad-b158-444f-8406-25d8f70e9ec3, DELETED < Exit [DeferCleanup (Each)] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:96 @ 07/15/26 04:36:34.952 (176ms) > Enter [ReportAfterEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:99 @ 07/15/26 04:36:34.952 < Exit [ReportAfterEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:99 @ 07/15/26 04:36:34.952 (0s) > Enter [ReportAfterEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:155 @ 07/15/26 04:36:34.952 < Exit [ReportAfterEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:155 @ 07/15/26 04:36:34.952 (0s) 2026/07/15 04:37:04 Test passed > Enter [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/15/26 04:34:28.591 Create an experiment with name E2EExperiment-1784090068591482904 < Exit [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/15/26 04:34:28.801 (211ms) > Enter [BeforeEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:49 @ 07/15/26 04:34:28.802 ################### Setup before each Pipeline Upload test ##################### ################### Global Setup before each test ##################### Test Context: 0x23c4cbbe8500 Looks like test-secret-1 already exists, because creation failed due to secrets "test-secret-1" already exists Looks like test-secret-2 already exists, because creation failed due to secrets "test-secret-2" already exists Looks like test-secret-3 already exists, because creation failed due to secrets "test-secret-3" already exists < Exit [BeforeEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:49 @ 07/15/26 04:34:28.939 (137ms) > Enter [It] Upload essential/iris_pipeline_compiled.yaml pipeline - /dspa/backend/test/end2end/pipeline_e2e_test.go:152 @ 07/15/26 04:34:28.939 Uploading pipeline file essential/iris_pipeline_compiled.yaml Creating temp pipeline file with overridden SDK Version Uploading pipeline with name=e2e-test-1784090068802276430, from file /dspa/test_data/sdk_compiled_pipelines/valid/essential/iris_pipeline_compiled.yaml Upload of pipeline file 'essential/iris_pipeline_compiled.yaml' successful Listing pipeline versions for pipeline d275efb7-009d-44ec-86ee-33c6ba06e438 Get the pipeline run time inputs from pipeline spec file /dspa/test_data/sdk_compiled_pipelines/valid/essential/iris_pipeline_compiled.yaml Returning pipeline run time inputs map[] Create run for pipeline with id: 'd275efb7-009d-44ec-86ee-33c6ba06e438' and name: 'e2e-test-1784090068802276430' Create a pipeline run for pipeline with id=d275efb7-009d-44ec-86ee-33c6ba06e438 and versionId=f574f704-32ee-4bbb-94f9-e3505df8b042 Create a pipeline run body Created Pipeline Run successfully with runId=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Created Pipeline Run with id: cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 for pipeline with id: d275efb7-009d-44ec-86ee-33c6ba06e438 Waiting for pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 to be in one of '[SUCCEEDED SKIPPED FAILED CANCELED]' Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in PENDING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 is in RUNNING state, waiting... Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 reached expected state SUCCEEDED Deserializing expected compiled workflow file 'essential/iris_pipeline_compiled.yaml' for the pipeline Unmarshalling Expected Workflow YAML Unmarshalled Expected Workflow YAML Fetching updated pipeline run details for run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Get a pipeline run with id=cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Updated pipeline run details Pipeline run succeeded, checking if the number of tasks are what is expected < Exit [It] Upload essential/iris_pipeline_compiled.yaml pipeline - /dspa/backend/test/end2end/pipeline_e2e_test.go:152 @ 07/15/26 04:37:04.672 (2m35.733s) > Enter [DeferCleanup (Each)] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:96 @ 07/15/26 04:37:04.672 ################### Global Cleanup after each test ##################### Deleting 1 run(s) Terminate run cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Failed to terminate run cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Terminate run cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Deleting run cd35f82b-eec2-4ddb-8e60-7fd3e37475f1 Deleting 0 experiment(s) Deleting 1 pipeline(s) Deleting pipeline with id=d275efb7-009d-44ec-86ee-33c6ba06e438 (cascade=true) Pipeline with id=d275efb7-009d-44ec-86ee-33c6ba06e438, DELETED < Exit [DeferCleanup (Each)] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:96 @ 07/15/26 04:37:04.929 (257ms) > Enter [ReportAfterEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:99 @ 07/15/26 04:37:04.929 < Exit [ReportAfterEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:99 @ 07/15/26 04:37:04.93 (0s) > Enter [ReportAfterEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:155 @ 07/15/26 04:37:04.93 < Exit [ReportAfterEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:155 @ 07/15/26 04:37:04.93 (0s) 2026/07/15 04:37:45 Test passed > Enter [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/15/26 04:34:28.591 Create an experiment with name E2EExperiment-1784090068591079454 < Exit [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/15/26 04:34:28.798 (208ms) > Enter [BeforeEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:49 @ 07/15/26 04:34:28.798 ################### Setup before each Pipeline Upload test ##################### ################### Global Setup before each test ##################### Test Context: 0x50bab828100 test-secret-1 created test-secret-2 created test-secret-3 created < Exit [BeforeEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:49 @ 07/15/26 04:34:28.864 (66ms) > Enter [It] Upload critical/pipeline_with_artifact_upload_download.yaml pipeline - /dspa/backend/test/end2end/pipeline_e2e_test.go:152 @ 07/15/26 04:34:28.865 Uploading pipeline file critical/pipeline_with_artifact_upload_download.yaml Creating temp pipeline file with overridden SDK Version Uploading pipeline with name=e2e-test-1784090068798944091, from file /dspa/test_data/sdk_compiled_pipelines/valid/critical/pipeline_with_artifact_upload_download.yaml Upload of pipeline file 'critical/pipeline_with_artifact_upload_download.yaml' successful Listing pipeline versions for pipeline 852548b3-aa59-4118-9299-c71892adc550 Get the pipeline run time inputs from pipeline spec file /dspa/test_data/sdk_compiled_pipelines/valid/critical/pipeline_with_artifact_upload_download.yaml Returning pipeline run time inputs map[] Create run for pipeline with id: '852548b3-aa59-4118-9299-c71892adc550' and name: 'e2e-test-1784090068798944091' Create a pipeline run for pipeline with id=852548b3-aa59-4118-9299-c71892adc550 and versionId=1da5c62a-5efe-4899-ba77-c7ca46a8e60b Create a pipeline run body Created Pipeline Run successfully with runId=2d2b3bb1-7f45-434a-a807-250cd3d10f7f Created Pipeline Run with id: 2d2b3bb1-7f45-434a-a807-250cd3d10f7f for pipeline with id: 852548b3-aa59-4118-9299-c71892adc550 Waiting for pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f to be in one of '[SUCCEEDED SKIPPED FAILED CANCELED]' Get a pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f Pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f is in PENDING state, waiting... Get a pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f Pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f is in RUNNING state, waiting... Get a pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f Pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f is in RUNNING state, waiting... Get a pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f Pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f is in RUNNING state, waiting... Get a pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f Pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f is in RUNNING state, waiting... Get a pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f Pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f is in RUNNING state, waiting... Get a pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f Pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f is in RUNNING state, waiting... Get a pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f Pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f is in RUNNING state, waiting... Get a pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f Pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f is in RUNNING state, waiting... Get a pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f Pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f is in RUNNING state, waiting... Get a pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f Pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f is in RUNNING state, waiting... Get a pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f Pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f is in RUNNING state, waiting... Get a pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f Pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f is in RUNNING state, waiting... Get a pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f Pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f is in RUNNING state, waiting... Get a pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f Pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f is in RUNNING state, waiting... Get a pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f Pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f is in RUNNING state, waiting... Get a pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f Pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f is in RUNNING state, waiting... Get a pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f Pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f is in RUNNING state, waiting... Get a pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f Pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f is in RUNNING state, waiting... Get a pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f Pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f is in RUNNING state, waiting... Get a pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f Pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f reached expected state FAILED Deserializing expected compiled workflow file 'critical/pipeline_with_artifact_upload_download.yaml' for the pipeline Unmarshalling Expected Workflow YAML Unmarshalled Expected Workflow YAML Fetching updated pipeline run details for run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f Get a pipeline run with id=2d2b3bb1-7f45-434a-a807-250cd3d10f7f Updated pipeline run details Looks like the run 2d2b3bb1-7f45-434a-a807-250cd3d10f7f FAILED, so capture pod logs for the failed task FAILED - Task print-dataset-info for Run 2d2b3bb1-7f45-434a-a807-250cd3d10f7f did not complete successfully Capturing pod logs for task print-dataset-info, with pod name pipeline-with-datasets-jbc84-2663155168 Pod logs captured for task print-dataset-info in pod pipeline-with-datasets-jbc84-2663155168 Attaching pod logs to the report Failing 'print-dataset-info' Component Log - /dspa/backend/test/end2end/utils/e2e_utils.go:132 @ 07/15/26 04:36:09.655 2026-07-15T04:35:46.269701381Z time="2026-07-15T04:35:46.269Z" level=info msg="Starting Workflow Executor" version=v0.0.0+unknown 2026-07-15T04:35:46.272811305Z time="2026-07-15T04:35:46.272Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5 2026-07-15T04:35:46.272855122Z time="2026-07-15T04:35:46.272Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=dspa-test-rvemm250 podName=pipeline-with-datasets-jbc84-system-container-impl-2663155168 templateName=system-container-impl version="&Version{Version:v0.0.0+unknown,BuildDate:1970-01-01T00:00:00Z,GitCommit:,GitTag:,GitTreeState:,GoVersion:go1.24.6 (Red Hat 1.24.6-1.el9_6),Compiler:gc,Platform:linux/amd64,}" 2026-07-15T04:35:46.297836698Z time="2026-07-15T04:35:46.297Z" level=info msg="Starting deadline monitor" 2026-07-15T04:35:58.301356528Z time="2026-07-15T04:35:58.301Z" level=info msg="Main container completed" error="<nil>" 2026-07-15T04:35:58.301356528Z time="2026-07-15T04:35:58.301Z" level=info msg="No Script output reference in workflow. Capturing script output ignored" 2026-07-15T04:35:58.301356528Z time="2026-07-15T04:35:58.301Z" level=info msg="No output parameters" 2026-07-15T04:35:58.301388265Z time="2026-07-15T04:35:58.301Z" level=info msg="No output artifacts" 2026-07-15T04:35:58.312540869Z time="2026-07-15T04:35:58.312Z" level=info msg="Alloc=9789 TotalAlloc=20133 Sys=32597 NumGC=5 Goroutines=7" 2026-07-15T04:35:58.323484537Z time="2026-07-15T04:35:58.323Z" level=info msg="Deadline monitor stopped" 2026-07-15T04:35:46.462058253Z I0715 04:35:46.461913 21 main.go:66] Setting log level to: '1' 2026-07-15T04:35:46.483794053Z I0715 04:35:46.483723 21 cache.go:89] Connecting to cache endpoint ds-pipeline-dspa.dspa-test-rvemm250.svc.cluster.local:8887 2026-07-15T04:35:56.590760347Z I0715 04:35:56.590708 21 launcher_v2.go:202] publish success. 2026-07-15T04:35:56.636718375Z I0715 04:35:56.636664 21 client.go:797] Attempting to update DAG state 2026-07-15T04:35:56.721057168Z F0715 04:35:56.720998 21 main.go:58] failed to execute component: Failed to open bucket "mlpipeline": Failed to retrieve credentials for bucket mlpipeline: Failed to get Bucket credentials from secret name="ds-pipeline-s3-dspa" namespace="dspa-test-rvemm250": Get "https://172.30.0.1:443/api/v1/namespaces/dspa-test-rvemm250/secrets/ds-pipeline-s3-dspa": net/http: TLS handshake timeout 2026-07-15T04:35:57.438611585Z time="2026-07-15T04:35:57.438Z" level=info msg="sub-process exited" argo=true error="<nil>" 2026-07-15T04:35:57.439820579Z Error: exit status 1 Attached pod logs to the report FAILED - Task root for Run 2d2b3bb1-7f45-434a-a807-250cd3d10f7f did not complete successfully Capturing pod logs for task root, with pod name pipeline-with-datasets-jbc84-1449553430 Pod logs captured for task root in pod pipeline-with-datasets-jbc84-1449553430 Attaching pod logs to the report Failing 'root' Component Log - /dspa/backend/test/end2end/utils/e2e_utils.go:132 @ 07/15/26 04:36:09.739 2026-07-15T04:34:42.105153006Z time="2026-07-15T04:34:42.105Z" level=info msg="Starting Workflow Executor" version=v0.0.0+unknown 2026-07-15T04:34:42.109945662Z time="2026-07-15T04:34:42.109Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5 2026-07-15T04:34:42.109992674Z time="2026-07-15T04:34:42.109Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=dspa-test-rvemm250 podName=pipeline-with-datasets-jbc84-system-container-driver-1449553430 templateName=system-container-driver version="&Version{Version:v0.0.0+unknown,BuildDate:1970-01-01T00:00:00Z,GitCommit:,GitTag:,GitTreeState:,GoVersion:go1.24.6 (Red Hat 1.24.6-1.el9_6),Compiler:gc,Platform:linux/amd64,}" 2026-07-15T04:34:42.146501664Z time="2026-07-15T04:34:42.146Z" level=info msg="Starting deadline monitor" 2026-07-15T04:34:45.147560219Z time="2026-07-15T04:34:45.147Z" level=info msg="Main container completed" error="<nil>" 2026-07-15T04:34:45.147560219Z time="2026-07-15T04:34:45.147Z" level=info msg="No Script output reference in workflow. Capturing script output ignored" 2026-07-15T04:34:45.147560219Z time="2026-07-15T04:34:45.147Z" level=info msg="Saving output parameters" 2026-07-15T04:34:45.147560219Z time="2026-07-15T04:34:45.147Z" level=info msg="Saving path output parameter: pod-spec-patch" 2026-07-15T04:34:45.147560219Z time="2026-07-15T04:34:45.147Z" level=info msg="Copying /tmp/outputs/pod-spec-patch from base image layer" 2026-07-15T04:34:45.147632016Z time="2026-07-15T04:34:45.147Z" level=info msg="Successfully saved output parameter: pod-spec-patch" 2026-07-15T04:34:45.147632016Z time="2026-07-15T04:34:45.147Z" level=info msg="Saving path output parameter: cached-decision" 2026-07-15T04:34:45.147641079Z time="2026-07-15T04:34:45.147Z" level=info msg="Copying /tmp/outputs/cached-decision from base image layer" 2026-07-15T04:34:45.147667183Z time="2026-07-15T04:34:45.147Z" level=info msg="Successfully saved output parameter: cached-decision" 2026-07-15T04:34:45.147667183Z time="2026-07-15T04:34:45.147Z" level=info msg="Saving path output parameter: condition" 2026-07-15T04:34:45.147674890Z time="2026-07-15T04:34:45.147Z" level=info msg="Copying /tmp/outputs/condition from base image layer" 2026-07-15T04:34:45.147697914Z time="2026-07-15T04:34:45.147Z" level=info msg="Successfully saved output parameter: condition" 2026-07-15T04:34:45.147697914Z time="2026-07-15T04:34:45.147Z" level=info msg="No output artifacts" 2026-07-15T04:34:45.162691580Z time="2026-07-15T04:34:45.162Z" level=info msg="Alloc=14525 TotalAlloc=20146 Sys=28245 NumGC=4 Goroutines=7" 2026-07-15T04:34:42.338552177Z I0715 04:34:42.338452 22 main.go:106] Setting log level to: '1' 2026-07-15T04:34:42.338552177Z I0715 04:34:42.338524 22 main.go:166] input ComponentSpec:{ 2026-07-15T04:34:42.338552177Z "executorLabel": "exec-download-dataset-and-upload-as-artifact", 2026-07-15T04:34:42.338552177Z "inputDefinitions": { 2026-07-15T04:34:42.338552177Z "parameters": { 2026-07-15T04:34:42.338552177Z "dataset_name": { 2026-07-15T04:34:42.338552177Z "parameterType": "STRING" 2026-07-15T04:34:42.338552177Z }, 2026-07-15T04:34:42.338552177Z "dataset_repo": { 2026-07-15T04:34:42.338552177Z "parameterType": "STRING" 2026-07-15T04:34:42.338552177Z } 2026-07-15T04:34:42.338552177Z } 2026-07-15T04:34:42.338552177Z }, 2026-07-15T04:34:42.338552177Z "outputDefinitions": { 2026-07-15T04:34:42.338552177Z "artifacts": { 2026-07-15T04:34:42.338552177Z "output_dataset": { 2026-07-15T04:34:42.338552177Z "artifactType": { 2026-07-15T04:34:42.338552177Z "schemaTitle": "system.Dataset", 2026-07-15T04:34:42.338552177Z "schemaVersion": "0.0.1" 2026-07-15T04:34:42.338552177Z } 2026-07-15T04:34:42.338552177Z } 2026-07-15T04:34:42.338552177Z } 2026-07-15T04:34:42.338552177Z } 2026-07-15T04:34:42.338552177Z } 2026-07-15T04:34:42.338944360Z I0715 04:34:42.338925 22 main.go:173] input TaskSpec:{ 2026-07-15T04:34:42.338944360Z "cachingOptions": { 2026-07-15T04:34:42.338944360Z "enableCache": true 2026-07-15T04:34:42.338944360Z }, 2026-07-15T04:34:42.338944360Z "componentRef": { 2026-07-15T04:34:42.338944360Z "name": "comp-download-dataset-and-upload-as-artifact" 2026-07-15T04:34:42.338944360Z }, 2026-07-15T04:34:42.338944360Z "inputs": { 2026-07-15T04:34:42.338944360Z "parameters": { 2026-07-15T04:34:42.338944360Z "dataset_name": { 2026-07-15T04:34:42.338944360Z "componentInputParameter": "dataset_name" 2026-07-15T04:34:42.338944360Z }, 2026-07-15T04:34:42.338944360Z "dataset_repo": { 2026-07-15T04:34:42.338944360Z "componentInputParameter": "dataset_repo" 2026-07-15T04:34:42.338944360Z } 2026-07-15T04:34:42.338944360Z } 2026-07-15T04:34:42.338944360Z }, 2026-07-15T04:34:42.338944360Z "taskInfo": { 2026-07-15T04:34:42.338944360Z "name": "download-dataset-and-upload-as-artifact" 2026-07-15T04:34:42.338944360Z } 2026-07-15T04:34:42.338944360Z } 2026-07-15T04:34:42.339256433Z I0715 04:34:42.339235 22 main.go:179] input ContainerSpec:{ 2026-07-15T04:34:42.339256433Z "args": [ 2026-07-15T04:34:42.339256433Z "--executor_input", 2026-07-15T04:34:42.339256433Z "{{$}}", 2026-07-15T04:34:42.339256433Z "--function_to_execute", 2026-07-15T04:34:42.339256433Z "download_dataset_and_upload_as_artifact" 2026-07-15T04:34:42.339256433Z ], 2026-07-15T04:34:42.339256433Z "command": [ 2026-07-15T04:34:42.339256433Z "sh", 2026-07-15T04:34:42.339256433Z "-c", 2026-07-15T04:34:42.339256433Z "\nif ! [ -x \"$(command -v pip)\" ]; then\n python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'datasets==4.0.0' \u0026\u0026 python3 -m pip install --quiet --no-warn-script-location 'git+https://github.com/opendatahub-io/data-science-pipelines.git@master#subdirectory=sdk/python' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"' \u0026\u0026 \"$0\" \"$@\"\n", 2026-07-15T04:34:42.339256433Z "sh", 2026-07-15T04:34:42.339256433Z "-ec", 2026-07-15T04:34:42.339256433Z "program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main --component_module_path \"$program_path/ephemeral_component.py\" \"$@\"\n", 2026-07-15T04:34:42.339256433Z "\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef download_dataset_and_upload_as_artifact(dataset_repo: str, dataset_name: str, output_dataset: Output[Dataset]):\n from datasets import load_dataset\n # Load data set from hugging face\n ds = load_dataset(dataset_repo, dataset_name)\n print(\"Downloaded Hugging Face data\")\n print(f\"Now saving to {output_dataset.path}\")\n ds.save_to_disk(output_dataset.path)\n print(f\"Saved to {output_dataset.path}\")\n\n" 2026-07-15T04:34:42.339256433Z ], 2026-07-15T04:34:42.339256433Z "image": "registry.redhat.io/ubi9/python-312@sha256:e80ff3673c95b91f0dafdbe97afb261eab8244d7fd8b47e20ffcbcfee27fb168" 2026-07-15T04:34:42.339256433Z } 2026-07-15T04:34:42.432112557Z I0715 04:34:42.432046 22 cache.go:89] Connecting to cache endpoint ds-pipeline-dspa.dspa-test-rvemm250.svc.cluster.local:8887 2026-07-15T04:34:42.452212623Z I0715 04:34:42.452144 22 client.go:385] Pipeline Context: id:5 name:"pipeline-with-datasets" type_id:10 type:"system.Pipeline" create_time_since_epoch:1784090072806 last_update_time_since_epoch:1784090072806 2026-07-15T04:34:42.521245292Z I0715 04:34:42.521177 22 client.go:394] Pipeline Run Context: id:9 name:"2d2b3bb1-7f45-434a-a807-250cd3d10f7f" type_id:11 type:"system.PipelineRun" custom_properties:{key:"namespace" value:{string_value:"dspa-test-rvemm250"}} custom_properties:{key:"pipeline_root" value:{string_value:"s3://mlpipeline/pipeline-with-datasets/2d2b3bb1-7f45-434a-a807-250cd3d10f7f"}} custom_properties:{key:"resource_name" value:{string_value:"run-resource"}} custom_properties:{key:"store_session_info" value:{string_value:"{\"Provider\":\"s3\",\"Params\":{\"accessKeyKey\":\"accesskey\",\"disableSSL\":\"false\",\"endpoint\":\"http://minio-dspa.dspa-test-rvemm250.svc.cluster.local:9000\",\"forcePathStyle\":\"true\",\"fromEnv\":\"false\",\"maxRetries\":\"5\",\"region\":\"minio\",\"secretKeyKey\":\"secretkey\",\"secretName\":\"ds-pipeline-s3-dspa\"}}"}} create_time_since_epoch:1784090073114 last_update_time_since_epoch:1784090073114 2026-07-15T04:34:42.917488366Z I0715 04:34:42.917371 22 container.go:73] parent DAG: id:3 name:"run/2d2b3bb1-7f45-434a-a807-250cd3d10f7f" type_id:12 type:"system.DAGExecution" last_known_state:RUNNING custom_properties:{key:"display_name" value:{string_value:""}} custom_properties:{key:"inputs" value:{struct_value:{fields:{key:"dataset_name" value:{string_value:""}} fields:{key:"dataset_repo" value:{string_value:"google/frames-benchmark"}}}}} custom_properties:{key:"task_name" value:{string_value:""}} create_time_since_epoch:1784090073606 last_update_time_since_epoch:1784090073606 2026-07-15T04:34:43.117704848Z I0715 04:34:43.117606 22 resolve.go:130] parent DAG input parameters: map[dataset_name:string_value:"" dataset_repo:string_value:"google/frames-benchmark"], artifacts: map[] 2026-07-15T04:34:43.117704848Z I0715 04:34:43.117695 22 cache.go:121] Task {download-dataset-and-upload-as-artifact} enables cache 2026-07-15T04:34:43.417032681Z I0715 04:34:43.416927 22 container.go:185] Created execution: id:6 type_id:13 type:"system.ContainerExecution" last_known_state:RUNNING custom_properties:{key:"cache_fingerprint" value:{string_value:"9af2953dc950ab9852d679f1694b20f978f0fce36f6839874b7ff2b9f42c561e"}} custom_properties:{key:"display_name" value:{string_value:"download-dataset-and-upload-as-artifact"}} custom_properties:{key:"image" value:{string_value:""}} custom_properties:{key:"inputs" value:{struct_value:{fields:{key:"dataset_name" value:{string_value:""}} fields:{key:"dataset_repo" value:{string_value:"google/frames-benchmark"}}}}} custom_properties:{key:"namespace" value:{string_value:""}} custom_properties:{key:"parent_dag_id" value:{int_value:3}} custom_properties:{key:"pod_name" value:{string_value:""}} custom_properties:{key:"pod_uid" value:{string_value:""}} custom_properties:{key:"task_name" value:{string_value:"download-dataset-and-upload-as-artifact"}} create_time_since_epoch:1784090083316 last_update_time_since_epoch:1784090083316 2026-07-15T04:34:43.418541415Z I0715 04:34:43.418518 22 main.go:303] output execution.ID=6 2026-07-15T04:34:43.418790114Z I0715 04:34:43.418767 22 main.go:339] output podSpecPatch= 2026-07-15T04:34:43.418790114Z {"containers":[{"name":"main","image":"registry.redhat.io/ubi9/python-312@sha256:e80ff3673c95b91f0dafdbe97afb261eab8244d7fd8b47e20ffcbcfee27fb168","command":["/kfp-launcher/launch","--pipeline_name","pipeline-with-datasets","--run_id","2d2b3bb1-7f45-434a-a807-250cd3d10f7f","--execution_id","6","--executor_input","{\"inputs\":{\"parameterValues\":{\"dataset_name\":\"\", \"dataset_repo\":\"google/frames-benchmark\"}}, \"outputs\":{\"artifacts\":{\"executor-logs\":{\"artifacts\":[{\"name\":\"executor-logs\", \"type\":{\"schemaTitle\":\"system.Artifact\"}, \"uri\":\"s3://mlpipeline/pipeline-with-datasets/2d2b3bb1-7f45-434a-a807-250cd3d10f7f/download-dataset-and-upload-as-artifact/497f62b2-5028-4b30-ab55-4908638b496f/executor-logs\"}]}, \"output_dataset\":{\"artifacts\":[{\"name\":\"output_dataset\", \"type\":{\"schemaTitle\":\"system.Dataset\", \"schemaVersion\":\"0.0.1\"}, \"uri\":\"s3://mlpipeline/pipeline-with-datasets/2d2b3bb1-7f45-434a-a807-250cd3d10f7f/download-dataset-and-upload-as-artifact/497f62b2-5028-4b30-ab55-4908638b496f/output_dataset\"}]}}, \"outputFile\":\"/s3/mlpipeline/pipeline-with-datasets/2d2b3bb1-7f45-434a-a807-250cd3d10f7f/download-dataset-and-upload-as-artifact/497f62b2-5028-4b30-ab55-4908638b496f/output_metadata.json\"}}","--component_spec","{\"inputDefinitions\":{\"parameters\":{\"dataset_name\":{\"parameterType\":\"STRING\"}, \"dataset_repo\":{\"parameterType\":\"STRING\"}}}, \"outputDefinitions\":{\"artifacts\":{\"executor-logs\":{\"artifactType\":{\"schemaTitle\":\"system.Artifact\"}}, \"output_dataset\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\", \"schemaVersion\":\"0.0.1\"}}}}, \"executorLabel\":\"exec-download-dataset-and-upload-as-artifact\"}","--pod_name","$(KFP_POD_NAME)","--pod_uid","$(KFP_POD_UID)","--ml_pipeline_server_address","ds-pipeline-dspa.dspa-test-rvemm250.svc.cluster.local","--ml_pipeline_server_port","8887","--mlmd_server_address","ds-pipeline-metadata-grpc-dspa.dspa-test-rvemm250.svc.cluster.local","--mlmd_server_port","8080","--publish_logs","true","--ml_pipeline_tls_enabled","--metadata_tls_enabled","--ca_cert_path","/kfp/certs/ca.crt","--publish_logs","true","--"],"args":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'datasets==4.0.0' \u0026\u0026 python3 -m pip install --quiet --no-warn-script-location 'git+https://github.com/opendatahub-io/data-science-pipelines.git@master#subdirectory=sdk/python' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main --component_module_path \"$program_path/ephemeral_component.py\" \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef download_dataset_and_upload_as_artifact(dataset_repo: str, dataset_name: str, output_dataset: Output[Dataset]):\n from datasets import load_dataset\n # Load data set from hugging face\n ds = load_dataset(dataset_repo, dataset_name)\n print(\"Downloaded Hugging Face data\")\n print(f\"Now saving to {output_dataset.path}\")\n ds.save_to_disk(output_dataset.path)\n print(f\"Saved to {output_dataset.path}\")\n\n","--executor_input","{{$}}","--function_to_execute","download_dataset_and_upload_as_artifact"],"resources":{}}]} 2026-07-15T04:34:43.418907067Z I0715 04:34:43.418890 22 main.go:353] output ExecutorInput:{ 2026-07-15T04:34:43.418907067Z "inputs": { 2026-07-15T04:34:43.418907067Z "parameterValues": { 2026-07-15T04:34:43.418907067Z "dataset_name": "", 2026-07-15T04:34:43.418907067Z "dataset_repo": "google/frames-benchmark" 2026-07-15T04:34:43.418907067Z } 2026-07-15T04:34:43.418907067Z }, 2026-07-15T04:34:43.418907067Z "outputs": { 2026-07-15T04:34:43.418907067Z "artifacts": { 2026-07-15T04:34:43.418907067Z "executor-logs": { 2026-07-15T04:34:43.418907067Z "artifacts": [ 2026-07-15T04:34:43.418907067Z { 2026-07-15T04:34:43.418907067Z "name": "executor-logs", 2026-07-15T04:34:43.418907067Z "type": { 2026-07-15T04:34:43.418907067Z "schemaTitle": "system.Artifact" 2026-07-15T04:34:43.418907067Z }, 2026-07-15T04:34:43.418907067Z "uri": "s3://mlpipeline/pipeline-with-datasets/2d2b3bb1-7f45-434a-a807-250cd3d10f7f/download-dataset-and-upload-as-artifact/497f62b2-5028-4b30-ab55-4908638b496f/executor-logs" 2026-07-15T04:34:43.418907067Z } 2026-07-15T04:34:43.418907067Z ] 2026-07-15T04:34:43.418907067Z }, 2026-07-15T04:34:43.418907067Z "output_dataset": { 2026-07-15T04:34:43.418907067Z "artifacts": [ 2026-07-15T04:34:43.418907067Z { 2026-07-15T04:34:43.418907067Z "name": "output_dataset", 2026-07-15T04:34:43.418907067Z "type": { 2026-07-15T04:34:43.418907067Z "schemaTitle": "system.Dataset", 2026-07-15T04:34:43.418907067Z "schemaVersion": "0.0.1" 2026-07-15T04:34:43.418907067Z }, 2026-07-15T04:34:43.418907067Z "uri": "s3://mlpipeline/pipeline-with-datasets/2d2b3bb1-7f45-434a-a807-250cd3d10f7f/download-dataset-and-upload-as-artifact/497f62b2-5028-4b30-ab55-4908638b496f/output_dataset" 2026-07-15T04:34:43.418907067Z } 2026-07-15T04:34:43.418907067Z ] 2026-07-15T04:34:43.418907067Z } 2026-07-15T04:34:43.418907067Z }, 2026-07-15T04:34:43.418907067Z "outputFile": "/s3/mlpipeline/pipeline-with-datasets/2d2b3bb1-7f45-434a-a807-250cd3d10f7f/download-dataset-and-upload-as-artifact/497f62b2-5028-4b30-ab55-4908638b496f/output_metadata.json" 2026-07-15T04:34:43.418907067Z } 2026-07-15T04:34:43.418907067Z } 2026-07-15T04:34:44.312578902Z time="2026-07-15T04:34:44.312Z" level=info msg="sub-process exited" argo=true error="<nil>" 2026-07-15T04:34:44.312612933Z time="2026-07-15T04:34:44.312Z" level=info msg="/tmp/outputs/pod-spec-patch -> /var/run/argo/outputs/parameters//tmp/outputs/pod-spec-patch" argo=true 2026-07-15T04:34:44.312883070Z time="2026-07-15T04:34:44.312Z" level=info msg="/tmp/outputs/cached-decision -> /var/run/argo/outputs/parameters//tmp/outputs/cached-decision" argo=true 2026-07-15T04:34:44.312986820Z time="2026-07-15T04:34:44.312Z" level=info msg="/tmp/outputs/condition -> /var/run/argo/outputs/parameters//tmp/outputs/condition" argo=true Attached pod logs to the report FAILED - Task pipeline-with-datasets-jbc84 for Run 2d2b3bb1-7f45-434a-a807-250cd3d10f7f did not complete successfully Capturing pod logs for task pipeline-with-datasets-jbc84, with pod name pipeline-with-datasets-jbc84-3546502295 Pod logs captured for task pipeline-with-datasets-jbc84 in pod pipeline-with-datasets-jbc84-3546502295 Attaching pod logs to the report Failing 'pipeline-with-datasets-jbc84' Component Log - /dspa/backend/test/end2end/utils/e2e_utils.go:132 @ 07/15/26 04:36:09.841 2026-07-15T04:34:32.004242458Z time="2026-07-15T04:34:32.004Z" level=info msg="Starting Workflow Executor" version=v0.0.0+unknown 2026-07-15T04:34:32.007363465Z time="2026-07-15T04:34:32.007Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5 2026-07-15T04:34:32.007391705Z time="2026-07-15T04:34:32.007Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=dspa-test-rvemm250 podName=pipeline-with-datasets-jbc84-system-dag-driver-3546502295 templateName=system-dag-driver version="&Version{Version:v0.0.0+unknown,BuildDate:1970-01-01T00:00:00Z,GitCommit:,GitTag:,GitTreeState:,GoVersion:go1.24.6 (Red Hat 1.24.6-1.el9_6),Compiler:gc,Platform:linux/amd64,}" 2026-07-15T04:34:32.034532328Z time="2026-07-15T04:34:32.034Z" level=info msg="Starting deadline monitor" 2026-07-15T04:34:35.035470370Z time="2026-07-15T04:34:35.035Z" level=info msg="Main container completed" error="<nil>" 2026-07-15T04:34:35.035470370Z time="2026-07-15T04:34:35.035Z" level=info msg="No Script output reference in workflow. Capturing script output ignored" 2026-07-15T04:34:35.035470370Z time="2026-07-15T04:34:35.035Z" level=info msg="Saving output parameters" 2026-07-15T04:34:35.035470370Z time="2026-07-15T04:34:35.035Z" level=info msg="Saving path output parameter: execution-id" 2026-07-15T04:34:35.035470370Z time="2026-07-15T04:34:35.035Z" level=info msg="Copying /tmp/outputs/execution-id from base image layer" 2026-07-15T04:34:35.035515239Z time="2026-07-15T04:34:35.035Z" level=info msg="Successfully saved output parameter: execution-id" 2026-07-15T04:34:35.035515239Z time="2026-07-15T04:34:35.035Z" level=info msg="Saving path output parameter: iteration-count" 2026-07-15T04:34:35.035523795Z time="2026-07-15T04:34:35.035Z" level=info msg="Copying /tmp/outputs/iteration-count from base image layer" 2026-07-15T04:34:35.035549659Z time="2026-07-15T04:34:35.035Z" level=info msg="Successfully saved output parameter: iteration-count" 2026-07-15T04:34:35.035557644Z time="2026-07-15T04:34:35.035Z" level=info msg="Saving path output parameter: condition" 2026-07-15T04:34:35.035557644Z time="2026-07-15T04:34:35.035Z" level=info msg="Copying /tmp/outputs/condition from base image layer" 2026-07-15T04:34:35.035799254Z time="2026-07-15T04:34:35.035Z" level=info msg="Successfully saved output parameter: condition" 2026-07-15T04:34:35.035799254Z time="2026-07-15T04:34:35.035Z" level=info msg="No output artifacts" 2026-07-15T04:34:35.046372449Z time="2026-07-15T04:34:35.046Z" level=info msg="Alloc=9675 TotalAlloc=20153 Sys=33109 NumGC=5 Goroutines=7" 2026-07-15T04:34:32.342963699Z I0715 04:34:32.342848 20 main.go:106] Setting log level to: '1' 2026-07-15T04:34:32.343095399Z I0715 04:34:32.342944 20 main.go:166] input ComponentSpec:{ 2026-07-15T04:34:32.343095399Z "dag": { 2026-07-15T04:34:32.343095399Z "tasks": { 2026-07-15T04:34:32.343095399Z "download-dataset-and-upload-as-artifact": { 2026-07-15T04:34:32.343095399Z "cachingOptions": { 2026-07-15T04:34:32.343095399Z "enableCache": true 2026-07-15T04:34:32.343095399Z }, 2026-07-15T04:34:32.343095399Z "componentRef": { 2026-07-15T04:34:32.343095399Z "name": "comp-download-dataset-and-upload-as-artifact" 2026-07-15T04:34:32.343095399Z }, 2026-07-15T04:34:32.343095399Z "inputs": { 2026-07-15T04:34:32.343095399Z "parameters": { 2026-07-15T04:34:32.343095399Z "dataset_name": { 2026-07-15T04:34:32.343095399Z "componentInputParameter": "dataset_name" 2026-07-15T04:34:32.343095399Z }, 2026-07-15T04:34:32.343095399Z "dataset_repo": { 2026-07-15T04:34:32.343095399Z "componentInputParameter": "dataset_repo" 2026-07-15T04:34:32.343095399Z } 2026-07-15T04:34:32.343095399Z } 2026-07-15T04:34:32.343095399Z }, 2026-07-15T04:34:32.343095399Z "taskInfo": { 2026-07-15T04:34:32.343095399Z "name": "download-dataset-and-upload-as-artifact" 2026-07-15T04:34:32.343095399Z } 2026-07-15T04:34:32.343095399Z }, 2026-07-15T04:34:32.343095399Z "print-dataset-info": { 2026-07-15T04:34:32.343095399Z "cachingOptions": { 2026-07-15T04:34:32.343095399Z "enableCache": true 2026-07-15T04:34:32.343095399Z }, 2026-07-15T04:34:32.343095399Z "componentRef": { 2026-07-15T04:34:32.343095399Z "name": "comp-print-dataset-info" 2026-07-15T04:34:32.343095399Z }, 2026-07-15T04:34:32.343095399Z "dependentTasks": [ 2026-07-15T04:34:32.343095399Z "download-dataset-and-upload-as-artifact" 2026-07-15T04:34:32.343095399Z ], 2026-07-15T04:34:32.343095399Z "inputs": { 2026-07-15T04:34:32.343095399Z "artifacts": { 2026-07-15T04:34:32.343095399Z "dataset": { 2026-07-15T04:34:32.343095399Z "taskOutputArtifact": { 2026-07-15T04:34:32.343095399Z "outputArtifactKey": "output_dataset", 2026-07-15T04:34:32.343095399Z "producerTask": "download-dataset-and-upload-as-artifact" 2026-07-15T04:34:32.343095399Z } 2026-07-15T04:34:32.343095399Z } 2026-07-15T04:34:32.343095399Z } 2026-07-15T04:34:32.343095399Z }, 2026-07-15T04:34:32.343095399Z "taskInfo": { 2026-07-15T04:34:32.343095399Z "name": "print-dataset-info" 2026-07-15T04:34:32.343095399Z } 2026-07-15T04:34:32.343095399Z } 2026-07-15T04:34:32.343095399Z } 2026-07-15T04:34:32.343095399Z }, 2026-07-15T04:34:32.343095399Z "inputDefinitions": { 2026-07-15T04:34:32.343095399Z "parameters": { 2026-07-15T04:34:32.343095399Z "dataset_name": { 2026-07-15T04:34:32.343095399Z "defaultValue": "", 2026-07-15T04:34:32.343095399Z "isOptional": true, 2026-07-15T04:34:32.343095399Z "parameterType": "STRING" 2026-07-15T04:34:32.343095399Z }, 2026-07-15T04:34:32.343095399Z "dataset_repo": { 2026-07-15T04:34:32.343095399Z "defaultValue": "google/frames-benchmark", 2026-07-15T04:34:32.343095399Z "isOptional": true, 2026-07-15T04:34:32.343095399Z "parameterType": "STRING" 2026-07-15T04:34:32.343095399Z } 2026-07-15T04:34:32.343095399Z } 2026-07-15T04:34:32.343095399Z } 2026-07-15T04:34:32.343095399Z } 2026-07-15T04:34:32.344313560Z I0715 04:34:32.344289 20 main.go:179] input ContainerSpec:{} 2026-07-15T04:34:32.344360776Z I0715 04:34:32.344346 20 main.go:186] input RuntimeConfig:{ 2026-07-15T04:34:32.344360776Z "parameterValues": { 2026-07-15T04:34:32.344360776Z "dataset_name": "", 2026-07-15T04:34:32.344360776Z "dataset_repo": "google/frames-benchmark" 2026-07-15T04:34:32.344360776Z } 2026-07-15T04:34:32.344360776Z } 2026-07-15T04:34:32.344404350Z I0715 04:34:32.344390 20 main.go:292] input kubernetesConfig:{} 2026-07-15T04:34:32.423686171Z I0715 04:34:32.423622 20 cache.go:89] Connecting to cache endpoint ds-pipeline-dspa.dspa-test-rvemm250.svc.cluster.local:8887 2026-07-15T04:34:32.441704431Z I0715 04:34:32.441661 20 root_dag.go:100] PipelineRoot="s3://mlpipeline" from default config 2026-07-15T04:34:32.822819651Z I0715 04:34:32.822720 20 client.go:385] Pipeline Context: id:5 name:"pipeline-with-datasets" type_id:10 type:"system.Pipeline" create_time_since_epoch:1784090072806 last_update_time_since_epoch:1784090072806 2026-07-15T04:34:33.215212014Z I0715 04:34:33.215060 20 client.go:394] Pipeline Run Context: id:9 name:"2d2b3bb1-7f45-434a-a807-250cd3d10f7f" type_id:11 type:"system.PipelineRun" custom_properties:{key:"namespace" value:{string_value:"dspa-test-rvemm250"}} custom_properties:{key:"pipeline_root" value:{string_value:"s3://mlpipeline/pipeline-with-datasets/2d2b3bb1-7f45-434a-a807-250cd3d10f7f"}} custom_properties:{key:"resource_name" value:{string_value:"run-resource"}} custom_properties:{key:"store_session_info" value:{string_value:"{\"Provider\":\"s3\",\"Params\":{\"accessKeyKey\":\"accesskey\",\"disableSSL\":\"false\",\"endpoint\":\"http://minio-dspa.dspa-test-rvemm250.svc.cluster.local:9000\",\"forcePathStyle\":\"true\",\"fromEnv\":\"false\",\"maxRetries\":\"5\",\"region\":\"minio\",\"secretKeyKey\":\"secretkey\",\"secretName\":\"ds-pipeline-s3-dspa\"}}"}} create_time_since_epoch:1784090073114 last_update_time_since_epoch:1784090073114 2026-07-15T04:34:33.624429745Z I0715 04:34:33.624325 20 root_dag.go:133] Created execution: id:3 name:"run/2d2b3bb1-7f45-434a-a807-250cd3d10f7f" type_id:12 type:"system.DAGExecution" last_known_state:RUNNING custom_properties:{key:"display_name" value:{string_value:""}} custom_properties:{key:"inputs" value:{struct_value:{fields:{key:"dataset_name" value:{string_value:""}} fields:{key:"dataset_repo" value:{string_value:"google/frames-benchmark"}}}}} custom_properties:{key:"task_name" value:{string_value:""}} create_time_since_epoch:1784090073606 last_update_time_since_epoch:1784090073606 2026-07-15T04:34:33.624429745Z I0715 04:34:33.624405 20 main.go:303] output execution.ID=3 2026-07-15T04:34:34.317870409Z time="2026-07-15T04:34:34.317Z" level=info msg="sub-process exited" argo=true error="<nil>" 2026-07-15T04:34:34.317915043Z time="2026-07-15T04:34:34.317Z" level=info msg="/tmp/outputs/execution-id -> /var/run/argo/outputs/parameters//tmp/outputs/execution-id" argo=true 2026-07-15T04:34:34.318285805Z time="2026-07-15T04:34:34.318Z" level=info msg="/tmp/outputs/iteration-count -> /var/run/argo/outputs/parameters//tmp/outputs/iteration-count" argo=true 2026-07-15T04:34:34.318389812Z time="2026-07-15T04:34:34.318Z" level=info msg="/tmp/outputs/condition -> /var/run/argo/outputs/parameters//tmp/outputs/condition" argo=true Attached pod logs to the report FAILED - Task executor for Run 2d2b3bb1-7f45-434a-a807-250cd3d10f7f did not complete successfully SUCCEEDED - Task print-dataset-info-driver for run 2d2b3bb1-7f45-434a-a807-250cd3d10f7f has finished successfully SUCCEEDED - Task download-dataset-and-upload-as-artifact for run 2d2b3bb1-7f45-434a-a807-250cd3d10f7f has finished successfully SUCCEEDED - Task executor for run 2d2b3bb1-7f45-434a-a807-250cd3d10f7f has finished successfully SUCCEEDED - Task download-dataset-and-upload-as-artifact-driver for run 2d2b3bb1-7f45-434a-a807-250cd3d10f7f has finished successfully SUCCEEDED - Task root-driver for run 2d2b3bb1-7f45-434a-a807-250cd3d10f7f has finished successfully Found failed tasks: 0x2b63180 [FAILED] Failure recorded during attempt 1: Failing test because the pipeline run was not SUCCESSFUL In [It] at: /dspa/backend/test/end2end/utils/e2e_utils.go:87 @ 07/15/26 04:36:09.841 < Exit [It] Upload critical/pipeline_with_artifact_upload_download.yaml pipeline - /dspa/backend/test/end2end/pipeline_e2e_test.go:152 @ 07/15/26 04:36:09.841 (1m40.976s) > Enter [DeferCleanup (Each)] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:96 @ 07/15/26 04:36:09.841 ################### Global Cleanup after each test ##################### Deleting 1 run(s) Terminate run 2d2b3bb1-7f45-434a-a807-250cd3d10f7f Failed to terminate run 2d2b3bb1-7f45-434a-a807-250cd3d10f7f Terminate run 2d2b3bb1-7f45-434a-a807-250cd3d10f7f Deleting run 2d2b3bb1-7f45-434a-a807-250cd3d10f7f Deleting 0 experiment(s) Deleting 1 pipeline(s) Deleting pipeline with id=852548b3-aa59-4118-9299-c71892adc550 (cascade=true) Pipeline with id=852548b3-aa59-4118-9299-c71892adc550, DELETED < Exit [DeferCleanup (Each)] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:96 @ 07/15/26 04:36:10.099 (258ms) Attempt #1 Failed. Retrying ↺ @ 07/15/26 04:36:10.099 > Enter [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/15/26 04:36:10.1 Create an experiment with name E2EExperiment-1784090170100079772 < Exit [BeforeEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:147 @ 07/15/26 04:36:10.115 (15ms) > Enter [BeforeEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:49 @ 07/15/26 04:36:10.115 ################### Setup before each Pipeline Upload test ##################### ################### Global Setup before each test ##################### Test Context: 0x50bae830680 Looks like test-secret-1 already exists, because creation failed due to secrets "test-secret-1" already exists Looks like test-secret-2 already exists, because creation failed due to secrets "test-secret-2" already exists Looks like test-secret-3 already exists, because creation failed due to secrets "test-secret-3" already exists < Exit [BeforeEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:49 @ 07/15/26 04:36:10.5 (385ms) > Enter [It] Upload critical/pipeline_with_artifact_upload_download.yaml pipeline - /dspa/backend/test/end2end/pipeline_e2e_test.go:152 @ 07/15/26 04:36:10.5 Uploading pipeline file critical/pipeline_with_artifact_upload_download.yaml Creating temp pipeline file with overridden SDK Version Uploading pipeline with name=e2e-test-1784090170115648239, from file /dspa/test_data/sdk_compiled_pipelines/valid/critical/pipeline_with_artifact_upload_download.yaml Upload of pipeline file 'critical/pipeline_with_artifact_upload_download.yaml' successful Listing pipeline versions for pipeline 7e903e98-3f07-4f4e-b69f-8a6b477fffca Get the pipeline run time inputs from pipeline spec file /dspa/test_data/sdk_compiled_pipelines/valid/critical/pipeline_with_artifact_upload_download.yaml Returning pipeline run time inputs map[] Create run for pipeline with id: '7e903e98-3f07-4f4e-b69f-8a6b477fffca' and name: 'e2e-test-1784090170115648239' Create a pipeline run for pipeline with id=7e903e98-3f07-4f4e-b69f-8a6b477fffca and versionId=96e198e6-774a-4dd6-8c04-fb87c58347ac Create a pipeline run body Created Pipeline Run successfully with runId=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Created Pipeline Run with id: 7321b40e-a222-4a35-8e2b-3c5cad1b78b3 for pipeline with id: 7e903e98-3f07-4f4e-b69f-8a6b477fffca Waiting for pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 to be in one of '[SUCCEEDED SKIPPED FAILED CANCELED]' Get a pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 is in PENDING state, waiting... Get a pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 is in RUNNING state, waiting... Get a pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 is in RUNNING state, waiting... Get a pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 is in RUNNING state, waiting... Get a pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 is in RUNNING state, waiting... Get a pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 is in RUNNING state, waiting... Get a pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 is in RUNNING state, waiting... Get a pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 is in RUNNING state, waiting... Get a pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 is in RUNNING state, waiting... Get a pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 is in RUNNING state, waiting... Get a pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 is in RUNNING state, waiting... Get a pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 is in RUNNING state, waiting... Get a pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 is in RUNNING state, waiting... Get a pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 is in RUNNING state, waiting... Get a pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 is in RUNNING state, waiting... Get a pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 is in RUNNING state, waiting... Get a pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 is in RUNNING state, waiting... Get a pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 is in RUNNING state, waiting... Get a pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 is in RUNNING state, waiting... Get a pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 reached expected state SUCCEEDED Deserializing expected compiled workflow file 'critical/pipeline_with_artifact_upload_download.yaml' for the pipeline Unmarshalling Expected Workflow YAML Unmarshalled Expected Workflow YAML Fetching updated pipeline run details for run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Get a pipeline run with id=7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Updated pipeline run details Pipeline run succeeded, checking if the number of tasks are what is expected < Exit [It] Upload critical/pipeline_with_artifact_upload_download.yaml pipeline - /dspa/backend/test/end2end/pipeline_e2e_test.go:152 @ 07/15/26 04:37:45.646 (1m35.146s) > Enter [DeferCleanup (Each)] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:96 @ 07/15/26 04:37:45.647 ################### Global Cleanup after each test ##################### Deleting 1 run(s) Terminate run 7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Failed to terminate run 7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Terminate run 7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Deleting run 7321b40e-a222-4a35-8e2b-3c5cad1b78b3 Deleting 0 experiment(s) Deleting 1 pipeline(s) Deleting pipeline with id=7e903e98-3f07-4f4e-b69f-8a6b477fffca (cascade=true) Pipeline with id=7e903e98-3f07-4f4e-b69f-8a6b477fffca, DELETED < Exit [DeferCleanup (Each)] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:96 @ 07/15/26 04:37:45.802 (156ms) > Enter [ReportAfterEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:99 @ 07/15/26 04:37:45.802 < Exit [ReportAfterEach] Upload and Verify Pipeline Run > - /dspa/backend/test/end2end/pipeline_e2e_test.go:99 @ 07/15/26 04:37:45.802 (0s) > Enter [ReportAfterEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:155 @ 07/15/26 04:37:45.803 < Exit [ReportAfterEach] TOP-LEVEL - /dspa/backend/test/end2end/e2e_suite_test.go:155 @ 07/15/26 04:37:45.803 (0s)