Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Aspera CLI download support to pipeline #259

Merged
merged 10 commits into from
Jan 30, 2024
Merged

Add Aspera CLI download support to pipeline #259

merged 10 commits into from
Jan 30, 2024

Conversation

drpatelh
Copy link
Member

@drpatelh drpatelh commented Jan 29, 2024 •
edited
Loading

Closes #68

New parameter: --force_ftp_download to force FTP downloads.

I have set Aspera to be the new default for downloading data. I am not aware of any obvious limitations at this point with doing this so it feels right because it is lightning-fast compared to FTP and will allow users to report back issues quicker. We can revert this back to using FTP in the future quite quickly if needed.

I have run some full-sized tests on AWS and Azure comparing Aspera download times to FTP and we see a 50% speed-up in pipeline completion:

image

@github-actions GitHub Actions
Copy link

github-actions bot commented Jan 29, 2024 •
edited
Loading

nf-core lint overall result: Failed ❌

Posted for pipeline commit e33eef4

+| ✅ 147 tests passed       |+
#| ❔   9 tests were ignored |#
!| ❗   6 tests had warnings |!
-| ❌   1 tests failed       |-

❌ Test failures:

  • files_unchanged - assets/sendmail_template.txt does not match the template

❗ Test warnings:

  • files_exist - File not found: assets/multiqc_config.yml
  • files_exist - File not found: conf/igenomes.config
  • files_exist - File not found: .github/workflows/awstest.yml
  • files_exist - File not found: .github/workflows/awsfulltest.yml
  • files_exist - File not found: lib/WorkflowFetchngs.groovy
  • pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline

❔ Tests ignored:

  • files_exist - File is ignored: conf/modules.config
  • files_exist - File is ignored: lib/NfcoreTemplate.groovy
  • files_exist - File is ignored: lib/Utils.groovy
  • files_exist - File is ignored: lib/WorkflowMain.groovy
  • files_unchanged - File does not exist: lib/NfcoreTemplate.groovy
  • files_unchanged - File ignored due to lint config: .gitignore or .prettierignore or pyproject.toml
  • actions_ci - actions_ci
  • actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/fetchngs/fetchngs/.github/workflows/awstest.yml
  • multiqc_config - 'assets/multiqc_config.yml' not found

✅ Tests passed:

Run details

  • nf-core/tools version 2.12
  • Run at 2024-01-30 11:49:00

adamrtalbot
adamrtalbot reviewed Jan 30, 2024
View reviewed changes
Copy link
Contributor

@adamrtalbot adamrtalbot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code itself is fine.

But this is a major behavioural change of the pipeline, do we want to bundle it with the tags and template stuff? Given it's a new feature I'd want to add it to the next release.

workflows/sra/main.nf Show resolved Hide resolved
modules/local/aspera_cli/main.nf Show resolved Hide resolved
@drpatelh
Copy link
Member Author

drpatelh commented Jan 30, 2024 •
edited
Loading

But this is a major behavioural change of the pipeline, do we want to bundle it with the tags and template stuff?

Yep, this should be fine. We have a bunch of other bug fixes / features included in this release too. To be honest, not sure when we are going to get around to updating this pipeline again so may as well do it now. We just need to get it done now because it's been dragging.

@drpatelh drpatelh mentioned this pull request Jan 30, 2024
Open
@drpatelh drpatelh merged commit 80df440 into nf-core:dev Jan 30, 2024
38 of 40 checks passed
@drpatelh drpatelh deleted the aspera branch January 30, 2024 12:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@adamrtalbot adamrtalbot adamrtalbot approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@drpatelh @adamrtalbot