Data Ingestion Manager (Ingestion Web Interface)

This application is run under the assumption that EDEX services (qpidd, postgres, etc.) are already running.

How to set up the Data Ingestion Manager (DIM)

• Clone this repository python-ingestion in the home directory.
• Set up the virtual environment and config.py as outlined in the readme file for this repository.
• Ensure the virtual environment is activated.
• Set up the EDEX environment.
  • cd ~/python-ingestion/www/
  • cp edex-env.template edex-env
  • vim edex-env
    • Add the line export UFRAME_HOME=[uframe directory] where [uframe directory] is the current deployment of UFrame. Use the sample line as a reference to get the right directory (it should end with uframe-1.0 or something similar).
  • Save and close the file.
  • source edex-env.
• Install the web interface's required packages.
  • cd ~/python-ingestion/www/.
  • pip install -r requirements.txt.
• Create a PostgreSQL database called ingestion. This is the database that the web application will use. You must create this yourself.
• Set up the Django project settings.
  • cd ~/python-ingestion/www/src/diman/settings.
  • cp local.sample.env local.env.
  • vim local.env
  • Delete the line that contains DATABASE_URL=sqlite:///db.sqlite3.
  • Add the line DATABASE_URL=postgres://[username]:[password]@127.0.0.1:5432/ingestion, where [username]:[password] is the username and password for the PostgreSQL user.
    • For development/testing, any PostgreSQL user with access to the ingestion database should be fine.
    • For production, create a new user in PostgreSQL (e.g. u_ingestion) with a password.
    • The password for the user is stored as plain text in the local.env file.
  • Add the line DJANGO_SETTINGS_MODULE=diman.settings.development.
  • Save and close the file.
• Create the database tables and superuser.
  • cd ~/python-ingestion/www/src/.
  • python manage.py makemigrations
  • python manage.py migrate
  • python manage.py createsuperuser
    • Follow the prompts to create the superuser.

How to run the DIM in development mode

• Start a screen session or use multiple terminal windows to open three command prompts.
• In each terminal window, activate the virutalenv and source ~/python-ingestion/www/edex-env.
• Start the following processes in their own separate screens/terminals.
  • Django's development server.
    • cd ~/python-ingestion/www/src/.
    • python manage.py runserver_plus 0.0.0.0:8080
  • The Celery queue.
    • cd ~/python-ingestion/www/src/.
    • celery -A diman worker -l info --autoreload
  • The log watcher.
    • cd ~/python-ingestion/www/src/triage.
    • python log_watcher.py

How to use the DIM in development mode

• Point your browser to the address localhost:8080.
  • Click the LOG IN button in the upper right.
  • Log in using your previously created credentials.
  • Click the OOI Data Ingestion Management header to go back to the home page.
• Import a Deployment.
  • Click the "Create a Deployment" link.
  • Click the BROWSE button and load an ingestion CSV.
  • Click SAVE.
• Create an ingestion for the deployment.
  • On the Deployment's detail page, click ADD NEW INGESTION.
  • On the Create a New Ingestion page, override any options you need. Otherwise leave the boxes blank.
  • Click SUBMIT.
• Ingest data.
  • On the Ingestion's page, click "INGEST"
  • Data will begin ingesting in the Celery queue.