Migrating from old (kb dk) MerMEId install to the 2021 dev branch

Migrating from old (kb-dk) MerMEId install to the 2021 dev branch

These are notes from work to convert the Delius Catalogue of Works (published in 2018) for use using the latest (as of Jan 2021) Dockerised version of MerMEId on the Edirom GitHub repo development branch. [David Lewis]

Starting point

Ingredients:

• A complete export of the contents of the old eXist database (unzipped)
• A backup of the old Tomcat directory (I haven’t actually used this)
• Docker, and the Edirom MerMEId-dev docker image

Connecting to and managing the database and app

One source of confusion for me – I could not replace the default passwords by setting environment variables as explained in the install docs. This means that connection only works if you use default passwords. This affected my connection experiments below.

Management tools

• Java admin (XMLRPC):
  • The Java admin tools don’t work properly, and the MerMEId image doesn’t give admin access to the admin account. This may be because a) I used the wrong admin password (the one I set, not the default) or b) I'm using an incompatible version of Java admin tools. Currently, version 5.2 or 5.3 is required. (NB: to use Java admin tools, you need a non-docker install of eXist as well as the MerMEId one). I haven't retested this.
• WebDAV works at http://localhost:8080/webdav/db/ with default admin password. You can try using port 8443 and https, but it'll complain about certificates, and this is only localhost anyway.
• Exide does work at http://localhost:8080/apps/eXide/ (it also uses the default admin password)
  • Sometimes eXide can be a bit flaky. Different levels of reload/restart seem to be needed at times.

Accounts

Two accounts are created: mermeid and admin.

• admin can edit code and change files, but can't access web pages
• mermeid can access web pages, but can't edit code
• Since both are html authenticated, logging in to both means running two browsers (or private browsing)
• Zsophia suggests this could be resolved for me by giving admin access to mermaid: sm:add-group-member(mermedit, admin)

Step 1: Putting old data into new MerMEId

• Data was previously a set of MEI 3 files in /db/data-[collection], in our case /db/data-delius
• These MEI files must be converted using the MEI 3->MEI 4 XSLT
• The files can then be dropped into /db/apps/mermeid/data/
• Just doing this is enough to make the MerMEId editor work

Step 2: Re-publishing the catalogue

Challenges

• The sample publishing code has been removed from the repo (since it’s out of date)
• The old publishing code has a few issues:
  • It doesn’t use MEI 4
  • It uses old versions of some code that’s shared with the MerMEId editor
  • It has a dependency on a very old version of jquery [I need to check how hard that is to update]
  • It was designed for multiple collections – functionality that has been removed from the new version. This means:
    • Some paths can be simplified/hard-coded
    • There are some variables for the collection that aren’t needed (look for coll or c)

Process

• After converting and copying the data…
• Copy any extra RISM sigla (not yet tested), also abbreviations, keywords, etc.
• Resources (css, images, script, etc.) now have a more sensible directory structure, and old ones can be moved there, and paths updated accordingly [NB: recent discussions have suggested all project-specific files should be kept hygenically separated from MerMEId code to facilitate software updates (and general maintenance). We'll need some guidance for this]
• mei_to_html_public.xsl is the way the public web version overrides behaviour in the (MerMEId-core) mei_to_html.xsl. It includes the other file and adds to it. mei_to_html.xsl almost always has sensible defaults, but the public version needs updates.
  • The biggest changes here are:
    • workDesc → workList
    • work has more direct children (rather than via e.g. titleStmt)
    • Composer is now in work/contributor (formerly in respStmt)
    • Similarly the creation event is in the work, not in work/history
• The old web pages were XQuery files that called on modules (.xqm) for most of their functionality and html files for boilerplate content. The former need work:
  • These modules are needed for the MerMEId editor, but have diverged, including changes to function profiles. For now, I’m making copies of the files, duplicating functionality. These should be harmonised for sustainability's sake. It's not obvious that there should be much difference in functionality.
  • The MerMEId editor now uses $SESSION variables for allowing options to persist between pages, where previous versions used get variables in the URL. This makes cleaner code, but would require a GDPR cookie notice. For now, I’m making non-$session versions of them for published pages.
• I’m putting bespoke styling into separate CSS, where before it was incorporated (by me, carelessly)
• Previous versions of MerMEId required tricky web server configuration to handle URL rewriting. This is now handled by the MerMEId server in jetty-exist-additional-config/etc/webapp/WEB-INF/ in the repo. This covers request / file types and application plumbing, along with general mod_rewrite style things.
  • (enhancement request) The old .html → .xq mapping isn’t in there yet. If it were, external web server config would be minimal/non-existent
• Authentication is now handled by eXist, configured in the dockerfile (i.e. at creation of a docker image)
  • (enhancement request) This is great for the editor, but the published catalogue is unlikely to need or benefit from that. Having one dockerfile option with authentication disabled (but no exposed editing functionality) seems like a way of avoiding that.