Skip to content

Latest commit

 

History

History
404 lines (298 loc) · 15.1 KB

README.md

File metadata and controls

404 lines (298 loc) · 15.1 KB

docker-compose-rabbitmq-mysql

Overview

This repository illustrates a reference implementation of Senzing using RabbitMQ as the queue and MySQL as the underlying database.

The instructions show how to set up a system that:

  1. Reads JSON lines from a file on the internet.
  2. Sends each JSON line to a message queue.
    1. In this implementation, the queue is RabbitMQ.
  3. Reads messages from the queue and inserts into Senzing.
    1. In this implementation, Senzing keeps its data in a MySQL database.
  4. Reads information from Senzing via Senzing REST API server.
  5. Views resolved entities in a web app.

The following diagram shows the relationship of the docker containers in this docker composition. Arrows represent data flow.

Image of architecture

This docker formation brings up the following docker containers:

  1. bitnami/rabbitmq
  2. mysql
  3. phpmyadmin/phpmyadmin
  4. senzing/console
  5. senzing/entity-web-search-app
  6. senzing/init-container
  7. senzing/jupyter
  8. senzing/redoer
  9. senzing/stream-producer
  10. senzing/mysql-init
  11. senzing/senzing-api-server
  12. senzing/stream-loader

Contents

  1. Expectations
    1. Space
    2. Time
    3. Background knowledge
  2. Preparation
    1. Prerequisite software
    2. Clone repository
    3. Build docker images
  3. Using docker-compose
    1. Volumes
    2. SSH port
    3. Set sshd password
    4. EULA
    5. Install Senzing
    6. Install Senzing license
    7. Run docker formation
  4. View data
    1. View docker containers
    2. Use SSH
    3. View RabbitMQ
    4. View MySQL
    5. View Senzing API
    6. View Senzing Entity Search WebApp
    7. View Jupyter notebooks
    8. View X-Term
  5. Cleanup
  6. Advanced
    1. Re-run docker formation
    2. Configuration

Legend

  1. 🤔 - A "thinker" icon means that a little extra thinking may be required. Perhaps you'll need to make some choices. Perhaps it's an optional step.
  2. ✏️ - A "pencil" icon means that the instructions may need modification before performing.
  3. ⚠️ - A "warning" icon means that something tricky is happening, so pay attention.

Expectations

Space

This repository and demonstration require 7 GB free disk space.

Time

Budget 2 hours to get the demonstration up-and-running, depending on CPU and network speeds.

Background knowledge

This repository assumes a working knowledge of:

  1. Docker
  2. Docker-compose

Preparation

Prerequisite software

The following software programs need to be installed:

  1. docker
  2. docker-compose
  3. git

Clone repository

For more information on environment variables, see Environment Variables.

  1. Set these environment variable values:

    export GIT_ACCOUNT=senzing
export GIT_REPOSITORY=docker-compose-demo
export GIT_ACCOUNT_DIR=~/${GIT_ACCOUNT}.git
export GIT_REPOSITORY_DIR="${GIT_ACCOUNT_DIR}/${GIT_REPOSITORY}"

  2. Follow steps in clone-repository to install the Git repository.

Build docker images

  1. Build docker images.

    sudo docker build \
  --tag senzing/mysql-init \
  https://github.com/senzing/docker-mysql-init.git

Using docker-compose

Volumes

  1. ✏️ Specify the directory where Senzing should be installed on the local host. Example:

    export SENZING_VOLUME=/opt/my-senzing
    1. ⚠️ macOS - File sharing must be enabled for SENZING_VOLUME.
    2. ⚠️ Windows - File sharing must be enabled for SENZING_VOLUME.

  2. Identify directories on the local host. Example:

    export SENZING_DATA_DIR=${SENZING_VOLUME}/data
export SENZING_DATA_VERSION_DIR=${SENZING_DATA_DIR}/2.0.0
export SENZING_ETC_DIR=${SENZING_VOLUME}/etc
export SENZING_G2_DIR=${SENZING_VOLUME}/g2
export SENZING_VAR_DIR=${SENZING_VOLUME}/var

export MYSQL_DIR=${SENZING_VAR_DIR}/mysql
export RABBITMQ_DIR=${SENZING_VAR_DIR}/rabbitmq

  3. Create directory for RabbitMQ persistence. Note: Although the RABBITMQ_DIR directory will have open permissions, the directories created within RABBITMQ_DIR will be restricted. Example:

    sudo mkdir -p ${RABBITMQ_DIR}
sudo chmod 770 ${RABBITMQ_DIR}

SSH port

🤔 Optional If you do not plan on using the senzing/sshd container then these ssh sections can be ignored

🤔 Normally port 22 is already in use for ssh. So a different port may be needed by the running docker container.

  1. 🤔 Optional: See if port 22 is already in use. Example:

    sudo lsof -i -P -n | grep LISTEN | grep :22

  2. ✏️ Choose port for docker container. Example:

    export SENZING_SSHD_PORT=9181

  3. Construct parameter for docker run. Example:

    export SENZING_SSHD_PORT_PARAMETER="--publish ${SENZING_SSHD_PORT:-9181}:22"

Set sshd password

🤔 Optional The default password set for the sshd containers is senzingsshdpassword. However, this can be set by setting the following variable

✏️ Set the SENZING_SSHD_PASSWORD variable to change the password to access the sshd container

export SENZING_SSHD_PASSWORD=<Pass_You_Want>

EULA

To use the Senzing code, you must agree to the End User License Agreement (EULA).

  1. ⚠️ This step is intentionally tricky and not simply copy/paste. This ensures that you make a conscious effort to accept the EULA. Example:

    export SENZING_ACCEPT_EULA="<the value from this link>"

Install Senzing

  1. If Senzing has not been installed, install Senzing. Example:

    cd ${GIT_REPOSITORY_DIR}
sudo \
  --preserve-env \
  docker-compose --file resources/senzing/docker-compose-senzing-installation.yaml up
    1. This will download and extract a 3GB file. It may take 5-15 minutes, depending on network speeds.

Install Senzing license

Senzing comes with a trial license that supports 10,000 records.

  1. 🤔 Optional: If more than 10,000 records are desired, see Senzing license.

Run docker formation

  1. Launch docker-compose formation. Example:

    cd ${GIT_REPOSITORY_DIR}
sudo \
  --preserve-env \
  docker-compose --file resources/mysql/docker-compose-rabbitmq-mysql.yaml up

  2. Allow time for the components to come up and initialize.

    1. There will be errors in some docker logs as they wait for dependent services to become available. docker-compose isn't the best at orchestrating docker container dependencies.

View data

Once the docker-compose formation is running, different aspects of the formation can be viewed.

Username and password for the following sites were either passed in as environment variables or are the default values seen in docker-compose-rabbitmq-mysql.yaml.

View docker containers

  1. A good tool to monitor individual docker logs is Portainer. When running, Portainer is viewable at localhost:9170.

Use SSH

Instructions to use the senzing/sshd container are viewable in the senzing/docker-sshd repository

View RabbitMQ

  1. RabbitMQ is viewable at localhost:15672.
    1. Defaults: username: user password: bitnami
  2. See additional tips for working with RabbitMQ.

View MySQL

  1. MySQL is viewable at localhost:9173.
    1. Defaults: username: g2 password: g2
  2. See additional tips for working with MySQL.

View Senzing API

View results from Senzing REST API server. The server supports the Senzing REST API.

  1. OpenApi Editor is viewable at localhost:9180.
  2. Example Senzing REST API request: localhost:8250/heartbeat
  3. See additional tips for working with Senzing API server.

View Senzing Entity Search WebApp

  1. Senzing Entity Search WebApp is viewable at localhost:8251.
  2. See additional tips for working with Senzing Entity Search WebApp.

View Jupyter notebooks

  1. Change file permissions on MySQL database. Example:

    sudo chmod 777 -R ${SENZING_VAR_DIR}/mysql

  2. Jupyter Notebooks are viewable at localhost:9178.

  3. See additional tips for working with Jupyter Notebooks.

View X-Term

The web-based Senzing X-term can be used to run Senzing command-line programs.

  1. Senzing X-term is viewable at localhost:8254.
  2. See additional tips for working with Senzing X-Term.

Cleanup

When the docker-compose formation is no longer needed, it can be brought down and directories can be deleted.

  1. Bring down docker formation. Example:

    cd ${GIT_REPOSITORY_DIR}
sudo docker-compose --file resources/senzing/docker-compose-senzing-installation.yaml down
sudo docker-compose --file resources/mysql/docker-compose-rabbitmq-mysql.yaml down
sudo docker-compose --file resources/mysql/docker-compose-rabbitmq-mysql-again.yaml down

  2. Remove directories from host system. The following directories were created during the demonstration:

    1. ${SENZING_VOLUME}
    2. ${GIT_REPOSITORY_DIR}

    They may be safely deleted.

Advanced

The following topics discuss variations to the basic docker-compose demonstration.

Re-run docker formation

🤔 Optional: After the launch and shutdown of the original docker formation, the docker formation can be brought up again without requiring initialization steps. The following shows how to bring up the prior docker formation again without initialization.

  1. Launch docker-compose formation. Example:

    cd ${GIT_REPOSITORY_DIR}
sudo \
  --preserve-env \
  docker-compose --file resources/mysql/docker-compose-rabbitmq-mysql-again.yaml up

Configuration

Configuration values specified by environment variable or command line parameter.