Updating Movement for a New NIEM Release

The NIEM data model is constantly evolving. As new data is added and existing data is transformed, we must ensure that the Movement tool remains up-to-date. This wiki walks through the process of updating the DB/Search Engine (Solr) to a new NIEM version.

Download most recent version of NIEM from NIEM Releases GitHub Page. We will also be working with the movement-solr and movement-backend repos, so make sure you have checked these out and branched.

Copy the assets in the niem directory in the NIEM-Releases repo over to the niem-xsd directory in the movement-solr repo. It is safe to replace all contents.

In the movement-solr repo, open the data-config.xml file in the solr_config directory.

Here are the tasks we will perform in the data config update:

1. Update mapping of namespace prefixes
2. Update mapping of file and directory paths
3. Update list of elements that have no children

The data-config.xml file has several JavaScript functions that we will be updating.

First, find the getNameSpaceInfo function. Updating this function will complete the first two tasks mentioned above.

Update the domainLookup variable. The key should be the name of the directory and the value should be the full name of the domain. For example:

{
  cbrn: 'Chemical, Biological, Radiological, Nuclear',
  intelligence: 'Intelligence',
  ...
}

Next, update the namespaceLookup variable. They key should be the name of the directory inside of the over-arching niem-xsd directory. The value should be the namespace prefix. For example:

{
  'canada_post': 'can',
  'apco_event': 'apco',
  ...
}

At this point we have ingested all of the new release data with namespace and directory updates, but we still need to manually update the list of elements that have no children or substitution group heads, since we want to do this at index time.

Install docker and make sure you have installed and setup the niem-network:

docker network create niem-network

Build and run the Solr container, from the movement-solr repo's root directory:

docker build -t movement-solr .
docker run -dti -p 8983:8983 --name movement-solr --net niem-network movement-solr

Next, open the movement-backend repo.

To generate the new list of nonparentTypes and subGroupHeadIds for the Solr data import, run: node writeSolrIngestFiles. Note: Make sure the hostname in solrRequest.js points localhost.

Copy and paste the exports from the solrIngestData directory into the respective variables, nonparentTypeIds and subGroupHeaders in the data-config.xml file in the movement-solr repo.

Then re-build and run the solr docker container again.

docker rm -f movement-solr
docker build -t movement-solr .
docker run -dti -p 8983:8983 --name movement-solr --net niem-network movement-solr

The ingest should be complete! Before committing this change to the movement repos, test the data validity via the Solr GUI (localhost:8983) or via the movement web app if you have started that locally as well (recommended).