Harmony version 0.1.0

A second version of Harmony is in development as an API at https://github.com/harmonydata/harmony

Harmony is a data harmonisation project that uses Natural Language Processing to help researchers make better use of existing data from different studies by supporting them with the harmonisation of various measures and items used in different studies. Harmony is a collaboration project between the University of Ulster, University College London, the Universidade Federal de Santa Maria in Brazil, and Fast Data Science Ltd.

You can read more at https://harmonydata.org.

There is a live demo at: https://app.harmonydata.org/

This front end is based on the Dash Food Footprint demo: https://dash.gallery/dash-food-footprint/

Runs on Dash interactive Python framework developed by Plotly.

Developed by Thomas Wood / Fast Data Science thomas@fastdatascience.com

This tool is written in Python using the Dash front end library and the Java library Tika for reading PDFs, and runs on Linux, Mac, and Windows, and can be deployed as a web app using Docker.

How does Harmony work in layman's terms?

Harmony compares questions from different instruments by converting them to a vector representation and calculating their similarity. You can read more at https://harmonydata.org/how-does-harmony-work/

FAIR data schema

We have defined a data schema in accordance with the FAIR principles.

Questionnaires are represented within Harmony in a tabular format.

The file name is the unique identifier of a questionnaire, e.g. GAD-7 English.csv.

Files are tab-separated with the following columns:

• Question No: Alphanumeric, the question ID from the original questionnaire.
• Question: The text of the question
• Options: Any options or Likert scale such as "very often", "more than usual", etc

Very quick guide to running the tool on your computer

1. Install Docker.
2. Open a command line or Terminal window. Change folder to where you downloaded and unzipped the repository, and go to the folder front_end. Run the following command:

docker build -t harmony
docker run harmony

5. Open your browser at https://localhost:80. You will see the web app running.

Deploying the tool to Azure using the Azure Command Line Interface via Azure Container Registry

In command line, if you have installed Azure CLI, log into both the Azure Portal and Azure Container Registry:

az login
az acr login --name regprotocolsfds

If the admin user is not yet enabled, you can use the command:

az acr update -n regprotocolsfds --admin-enabled true

Run this script:

./build_deploy.sh

Developer's guide: Running the tool on your computer in Python and without using Docker

Architecture

Downloading PDF data

cd into data/raw_pdf and run download_raw_pdfs.sh.

Installing requirements

Download and install Java if you don't have it already. Download and install Apache Tika and run it on your computer https://tika.apache.org/download.html

java -jar tika-server-standard-2.3.0.jar

(the version number of your Jar file name may differ.)

Install everything in requirements.txt:

pip install -r requirements.txt

Running the front end app locally

Go into front_end and run

python application.py

You can then open your browser at localhost:8050 and you will see the tool.

Built With

• Dash - Main server and interactive components
• Plotly Python - Used to create the interactive plots
• Docker - Used for deployment to the web
• Apache Tika - Used for parsing PDFs to text
• spaCy - Used for NLP analysis
• NLTK - Used for NLP analysis
• Scikit-Learn - Used for machine learning

Licences of Third Party Software

• Apache Tika: Apache 2.0 License
• spaCy: MIT License
• NLTK: Apache 2.0 License
• Scikit-Learn: BSD 3-Clause

References

• Deploying a Dash webapp via Docker to Azure: https://medium.com/swlh/deploy-a-dash-application-in-azure-using-docker-ed46c4b9d2b2