New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Store lookup-tables in filesystem instead of loading via http #128

Open

dr0i opened this issue Feb 6, 2025 · 1 comment

Member

dr0i commented Feb 6, 2025

Tables that are used for the lookup via Fix are loaded via HTTP , e.g. https://raw.githubusercontent.com/hbz/lbz-vocabs/main/rpb-spatial.ttl. We could instead store the table in the filesystem (fs). Proposal: use gitsubmodules like we do in lobid-resources.

Atm, if one document is ETL-processed, at least ~10 MB of table is retrieved via HTTP.

Pro fs:

storing the tables in the filesystem acts like a cache:
faster
more reliable

Con fs:

there is no push mechanism of getting the data, i.e. if someone changes the data in th git repo of the lookup tables one has not immediatley the new data via gitsubmodules. At lobid-resources we update via gitsubmodules once a day resp. every time we transform.

The text was updated successfully, but these errors were encountered:

Contributor

TobiasNx commented Feb 6, 2025 •

edited

Loading

Wir können hier im RPB Repo auch deutsch schreiben, die Einbindung als git submodules muss zusätzlich in den fix transformationen angepasst werden.

Braucht es für die tests auch lokale Kopien, oder werden die Dateien beim gitsubmodul auch ausgecheckt, wenn man sich das RPB-Repo clont?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment