Skip to content

Speedup first time loading of RIT catalog #30

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
vaishakp opened this issue May 14, 2023 · 1 comment
Open

Speedup first time loading of RIT catalog #30

vaishakp opened this issue May 14, 2023 · 1 comment
Assignees
@prayush
Labels
enhancement New feature or request

Comments

@vaishakp
Copy link
Collaborator

I was wondering if its possible to speedup the first time download/ load of RIT catalog using parallel requests. As I understand, the catalog presently downloads the metadata serially. It takes about half a day to download the catalog.

Would it be possible to cut down this time using wget's parallel download (using a constructed list of urls) or multiple processes?
@vaishakp vaishakp changed the title Speedup first time loading of RIT catalog. Speedup first time loading of RIT catalog May 14, 2023
@prayush prayush added the enhancement New feature or request label May 19, 2023
@prayush prayush self-assigned this May 19, 2023
@adivijaykumar
Copy link
Collaborator

I have been using https://gist.github.com/adivijaykumar/8eb4ea175c03893ebdeb2b4e38a73a8a to wget all files to the cache directory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@prayush prayush

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@prayush @adivijaykumar @vaishakp