Skip to content

Latest commit

 

History

History
400 lines (286 loc) · 14.2 KB

README.md

File metadata and controls

400 lines (286 loc) · 14.2 KB

docker-compose-rabbitmq-postgresql-advanced

Synopsis

Using docker-compose, bring up a Senzing stack using RabbitMQ and a PostgreSQL database.

These instructions support multiple docker-compose.yaml files.

Overview

This repository illustrates a reference implementation of Senzing using RabbitMQ as the queue and PostgreSQL as the underlying database.

The instructions show how to set up a system that:

  1. Reads JSON lines from a file on the internet and sends each JSON line to a message queue via the Senzing stream-producer.
    1. In this implementation, the queue is RabbitMQ.
  2. Reads messages from the queue and inserts into Senzing via the Senzing stream-loader.
    1. In this implementation, Senzing keeps its data in a PostgreSQL database.
  3. Reads information from Senzing via Senzing API Server server.
  4. Views resolved entities in a web app.

The following diagram shows the relationship of the Docker containers in this Docker composition. Arrows represent data flow.

Image of architecture

Contents

  1. Prerequisites
  2. Demonstrate
    1. Choose formation
      1. Standard formation
      2. With Senzing API Server formation
      3. Withinfo formation
      4. Withinfo and Redoer formation
      5. Withinfo and Redoer queuing formation
      6. Debugging
  3. Advanced
  4. Related artifacts
  5. References

Preamble

At Senzing, we strive to create GitHub documentation in a "don't make me think" style. For the most part, instructions are copy and paste. Icons are used to signify additional actions by the user. If the instructions are not clear, please let us know by opening a new Documentation issue describing where we can improve. Now on with the show...

Expectations

  • Space: This repository and demonstration require 7 GB free disk space.
  • Time: Budget 2 hours to get the demonstration up-and-running, depending on CPU and network speeds.
  • Background knowledge: This repository assumes a working knowledge of:

Prerequisites

  1. docker - Minimum version: 20.10.16
  2. docker-compose - Minimum version: 1.29.0

Demonstrate

Choose formation

🤔 Choose a docker-compose.yaml file from list. Choose one value for SENZING_DOCKER_COMPOSE_FILE from the examples given below.

Standard formation

  1. Standard demonstration.

    export SENZING_DOCKER_COMPOSE_FILE=resources/postgresql/docker-compose-rabbitmq-postgresql.yaml

With Senzing API Server formation

Uses senzing/senzing-api-server instead of senzing/senzing-poc-server.

  1. Standard demonstration.

    export SENZING_DOCKER_COMPOSE_FILE=resources/postgresql/docker-compose-rabbitmq-postgresql-api-server.yaml

Withinfo formation

  1. Return information with each record added to Senzing.

    export SENZING_DOCKER_COMPOSE_FILE=resources/postgresql/docker-compose-rabbitmq-postgresql-withinfo.yaml

Withinfo and Redoer formation

  1. Add redoer to standard demonstration. Also, return information with each record added to Senzing.

    export SENZING_DOCKER_COMPOSE_FILE=resources/postgresql/docker-compose-rabbitmq-postgresql-redoer-withinfo.yaml

Withinfo and Redoer queuing formation

  1. Add multiple redoers to standard demonstration. This will process the Senzing "redo records". One redoer will populate RabbitMQ with redo records. One or more redoers will read redo records from RabbitMQ topic and send to the Senzing Engine. Also, return information with each record added to Senzing.

    export SENZING_DOCKER_COMPOSE_FILE=resources/postgresql/docker-compose-rabbitmq-postgresql-redoer-rabbitmq-withinfo.yaml

Debugging

  1. Run with SENZING_LOG_LEVEL=debug and --cap-add ALL in stream-loader and redoer containers. This will return pstack data in the log.

    export SENZING_DOCKER_COMPOSE_FILE=resources/postgresql/docker-compose-rabbitmq-postgresql-debug.yaml

Run formation

  1. ✏️ Specify a new directory to hold demonstration artifacts on the local host. Example:

    export SENZING_DEMO_DIR=~/my-senzing
    1. ⚠️ macOS - File sharing must be enabled for SENZING_DEMO_DIR.
    2. ⚠️ Windows - File sharing must be enabled for SENZING_DEMO_DIR.

  2. Set environment variables. Example:

    export PGADMIN_DIR=${SENZING_DEMO_DIR}/pgadmin
export POSTGRES_DIR=${SENZING_DEMO_DIR}/postgres
export RABBITMQ_DIR=${SENZING_DEMO_DIR}/rabbitmq
export SENZING_GID=$(id -g)
export SENZING_UID=$(id -u)
export SENZING_VAR_DIR=${SENZING_DEMO_DIR}/var

  3. Create directories. Example:

    mkdir -p ${PGADMIN_DIR} ${POSTGRES_DIR} ${RABBITMQ_DIR} ${SENZING_VAR_DIR}
chmod -R 777 ${SENZING_DEMO_DIR}

  4. Get versions of Docker images. Example:

    curl -X GET \
    --output ${SENZING_DEMO_DIR}/docker-versions-stable.sh \
    https://raw.githubusercontent.com/Senzing/knowledge-base/main/lists/docker-versions-stable.sh
source ${SENZING_DEMO_DIR}/docker-versions-stable.sh

  5. Download docker-compose.yaml and Docker images. Example:

    curl -X GET \
    --output ${SENZING_DEMO_DIR}/docker-compose.yaml \
    "https://raw.githubusercontent.com/senzing-garage/docker-compose-demo/main/${SENZING_DOCKER_COMPOSE_FILE}"
cd ${SENZING_DEMO_DIR}
sudo --preserve-env docker-compose pull

  6. Bring up Senzing docker-compose stack. Example:

    cd ${SENZING_DEMO_DIR}
sudo --preserve-env docker-compose up

  7. Allow time for the components to be downloaded, start, and initialize.

    1. There will be errors in some Docker logs as they wait for dependent services to become available. docker-compose isn't the best at orchestrating Docker container dependencies.

View formation

Once the docker-compose formation is running, different aspects of the formation can be viewed.

Username and password for the following sites were either passed in as environment variables or are the default values seen in ${SENZING_DOCKER_COMPOSE_FILE}.

View Docker containers

  1. A good tool to monitor individual Docker logs is Portainer. When running, Portainer is viewable at localhost:9170.

View Senzing Entity Search WebApp

  1. Senzing Entity Search WebApp is viewable at localhost:8251.
  2. See additional tips for working with Senzing Entity Search WebApp.

View X-Term

The web-based Senzing X-term can be used to run Senzing command-line programs.

  1. Senzing X-term is viewable at localhost:8254.
  2. See additional tips for working with Senzing X-Term.

Use SSH

Instructions to use the senzing/sshd container are viewable in the senzing/docker-sshd repository

View Jupyter notebooks

  1. Change file permissions on database files. Example:

    sudo chmod 777 -R ${POSTGRES_DIR}

  2. Jupyter Notebooks are viewable at localhost:9178.

  3. See additional tips for working with Jupyter Notebooks.

View Senzing API Server

View results from Senzing REST API server. The server supports the Senzing REST API.

  1. The OpenApi Editor with Servers value of http://localhost:8250 can be used to try the Senzing REST API.
  2. Example Senzing REST API request: localhost:8250/heartbeat
  3. See additional tips for working with Senzing API server.

View RabbitMQ

  1. RabbitMQ is viewable at localhost:15672.
    1. Defaults: username: user password: bitnami
  2. See additional tips for working with RabbitMQ.

View PostgreSQL

  1. PostgreSQL is viewable at localhost:9171.
    1. Database defaults: username: postgres password: postgres
  2. See additional tips for working with PostgreSQL.

Cleanup

When the docker-compose formation is no longer needed, it can be brought down and directories can be deleted.

  1. Bring down Docker formation. Example:

    cd ${SENZING_DEMO_DIR}
sudo docker-compose down

  2. Remove directories from host system. Example:

    rm -rf ${SENZING_DEMO_DIR:-/tmp/nowhere/for/safety}

Advanced

The following topics discuss variations to the basic docker-compose demonstration.

  1. SSH port
  2. Set sshd password

Docker images

This docker formation brings up the following docker containers:

  1. bitnami/postgres
  2. bitnami/rabbitmq
  3. busybox
  4. dpage/pgadmin4
  5. senzing/console
  6. senzing/entity-web-search-app
  7. senzing/jupyter
  8. senzing/redoer
  9. senzing/senzing-poc-server
  10. senzing/senzing-tools
  11. senzing/sshd
  12. senzing/stream-loader
  13. senzing/stream-producer
  14. senzing/xterm

Configuration

Configuration values specified by environment variable or command line parameter.

Program parameter matrix

  1. The matrix for using RabbitMQ with stream-loader.py and redoer.py subcommands.

    +-------------------------- stream-loader.py rabbitmq
|  +----------------------- stream-loader.py rabbitmq-withinfo
|  |  +-------------------- redoer.py redo
|  |  |  +----------------- redoer.py redo-withinfo-rabbitmq
|  |  |  |  +-------------- redoer.py write-to-rabbitmq
|  |  |  |  |  +----------- redoer.py read-from-rabbitmq
|  |  |  |  |  |  +-------- redoer.py read-from-rabbitmq-withinfo
|  |  |  |  |  |  |
v  v  v  v  v  v  v
X  .  .  .  .  .  .
X  .  X  .  .  .  .  docker-compose-rabbitmq-postgresql.yaml
X  .  .  X  .  .  .
X  .  .  .  X  X  .  docker-compose-rabbitmq-postgresql-redoer-rabbitmq.yaml
X  .  .  .  X  .  X
.  X  .  .  .  .  .  docker-compose-rabbitmq-postgresql-withinfo.yaml
.  X  X  .  .  .  .
.  X  .  X  .  .  .  docker-compose-rabbitmq-postgresql-redoer-withinfo.yaml
.  X  .  .  X  X  .
.  X  .  .  X  .  X  docker-compose-rabbitmq-postgresql-redoer-rabbitmq-withinfo.yaml

Related artifacts

  1. DockerHub

References