Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature]: Back up lists to the Internet Archive #131

Open
1 task done
T145 opened this issue Feb 15, 2024 · 3 comments
Open
1 task done

[feature]: Back up lists to the Internet Archive #131

T145 opened this issue Feb 15, 2024 · 3 comments
Assignees
Labels
enhancement Script or workflow edits to improve or add features

Comments

@T145
Copy link
Owner

T145 commented Feb 15, 2024

Contact Details

No response

What's your idea?

Instead of archiving lists to a local submodule, use the Internet Archive's API and libraries to backup lists and fall back to that URL if the original can't be accessed.

Code of Conduct

  • I agree to follow this project's Code of Conduct
@T145 T145 added the enhancement Script or workflow edits to improve or add features label Feb 15, 2024
@T145 T145 self-assigned this Feb 15, 2024
@T145
Copy link
Owner Author

T145 commented Apr 9, 2024

Prepend list URLs with https://web.archive.org/save/. Submitted 200 response means it gets saved successfully.

429: Save Page limits 15 requests per minute and will rate limit otherwise. This project should be fine.
509: Active session limit reached. Again this project should be fine.

@T145
Copy link
Owner Author

T145 commented May 1, 2024

Could detach the process so building can run alongside archival.

https://unix.stackexchange.com/questions/269805/how-can-i-detach-a-process-from-a-bash-script

The best plan is likely to perform archival before downloading the lists, catch any download errors, and retry those downloads with IA links.

@T145
Copy link
Owner Author

T145 commented Jul 7, 2024

(cb00c3f)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Script or workflow edits to improve or add features
Projects
None yet
Development

No branches or pull requests

1 participant