New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update gtdb-tk DM - chunck extractall for memory efficiency #6477

Merged

bgruening merged 1 commit into galaxyproject:main from paulzierep:update-gtdbtk-DM

Oct 21, 2024

Contributor

paulzierep commented Oct 21, 2024

FOR CONTRIBUTOR:

I have read the CONTRIBUTING.md document and this tool is appropriate for the tools-iuc repo.
License permits unrestricted use (educational + commercial)
This PR adds a new tool or tool collection
This PR updates an existing tool or tool collection
This PR does something else (explain below)

Thanks to usegalaxy.eu and @bgruening we know that this DM exceeds 100 GB memory usage. Probably because the tar.gz of the full DB includes huge files: https://github.com/paulzierep/tools-iuc/blob/bc882c3b2f21b4d3bda7a6c584121c8e3f95eaf0/data_managers/data_manager_gtdbtk_database_installer/test-data/release220/readme.md

Hopefully by chunking the files it can be reduced. Cannot really test it locally, though.


          chunck extractall for memory efficiency

1d60173

bgruening merged commit 1019bf0 into galaxyproject:main

11 checks passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet