Add cron job for lobid-organisation labels map

[TobiasNx]

src/main/resources/alma/maps/lobidOrganisationsMapping.tsv

Should be fetched via cron job.

https://lobid.org/organisations/search?q=_exists_%3Aisil+OR+type%3ACollection&format=csv:id,isil,sigel,name&size=25000

The process should fetch this and replace:
^" => empty
"$ => empty
"," => Tab
",," => 2x Tab

It seems that the request is forbidden in prod, but it works on test.

The columns are id, isil, sigel, name

This was asked by Digibib, since the labels shoud be updated regularly.

[dr0i]

It seems that the request is forbidden in prod, but it works on test.

I've now increased in the webserver config to timeout=60.

[TobiasNx]

thanks this seems to work now.

[TobiasNx]

Can you help with the seed script? For the cron job?

[dr0i]

Instead of regex the output it would be comfortable to have a tabulator-separated export in lobid-organisations => depends on hbz/lobid-organisations#500.

[TobiasNx]

https://lobid.org/organisations/search?q=_exists_%3Aisil+OR+type%3ACollection&format=tsv:id,isil,sigel,name&size=25000

[TobiasNx]

The new mechanism does not need any post processing. Only the following must be done:

1. Move the local file from src/main/resources/alma/maps/lobidOrganisationsMapping.tsv to src/test/resources/alma/maps
2. Cron job to create new src/main/resources/alma/maps/lobidOrganisationsMapping.tsv file with https://lobid.org/organisations/search?q=_exists_%3Aisil+OR+type%3ACollection&format=tsv:id,isil,sigel,name&size=25000

[TobiasNx]

Also adjust the links for the test here:

lobid-resources/src/main/java/org/lobid/resources/run/AlmaMarcXmlFix2lobidJsonEs.java

Lines 128 to 129 in 40f2c43

	fixVariables.put("lobidOrganisationsMapping.tsv", "./maps/lobidOrganisationsMapping.tsv");
	fixVariables.put("hbzowner2sigel.tsv", "./maps/hbzowner2sigel.tsv");

lobid-resources/src/test/java/org/lobid/resources/AlmaMarc21XmlToLobidJsonMetafixTest.java

Lines 72 to 73 in 40f2c43

	fixVariables.put("lobidOrganisationsMapping.tsv", "src/main/resources/alma/maps/lobidOrganisationsMapping.tsv");
	fixVariables.put("hbzowner2sigel.tsv", "src/main/resources/alma/maps/hbzowner2sigel.tsv");

[dr0i]

lobidOrganisationsMapping.tsv will be downloaded just before the automatical weekly deployment.