Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an "-resume" option for the bulk_downloader #31

Open
mberacochea opened this issue Oct 4, 2022 · 0 comments
Open

Add an "-resume" option for the bulk_downloader #31

mberacochea opened this issue Oct 4, 2022 · 0 comments
Labels
enhancement New feature or request

Comments

@mberacochea
Copy link
Member

MGnify has some very large studies, downloading those is problematic. With the current implementation if there is a network issue there is no way to restart the download process using the files already downloaded.

This feature will require (this is just a brain dump)

  • Store the tool progress status in a .sqlite db or a text file (pages and the download status for each page, how many pages...)
  • Add a "--resume" flag or sniff at the results folder before starting downloading data
  • Use the state to start downloading from that point
  • Check the files check the downloaded file checksums
@mberacochea mberacochea added the enhancement New feature or request label Oct 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant