Skip to content

Latest commit

 

History

History
417 lines (296 loc) · 14.6 KB

README.md

File metadata and controls

417 lines (296 loc) · 14.6 KB

docker-compose-sqs-postgresql-advanced

Synopsis

Using docker-compose, bring up a Senzing stack using AWS SQS and a PostgreSQL database.

These instructions support multiple docker-compose.yaml files.

Overview

This repository illustrates a reference implementation of Senzing using AWS SQS as the queue and PostgreSQL as the underlying database.

The instructions show how to set up a system that:

  1. Reads JSON lines from a file on the internet and sends each JSON line to a message queue via the Senzing stream-producer.
    1. In this implementation, the queue is AWS SQS.
  2. Reads messages from the queue and inserts into Senzing via the Senzing stream-loader.
    1. In this implementation, Senzing keeps its data in a PostgreSQL database.
  3. Reads information from Senzing via Senzing API Server server.
  4. Views resolved entities in a web app.

The following diagram shows the relationship of the Docker containers in this Docker composition. Arrows represent data flow.

Image of architecture

Contents

  1. Prerequisites
  2. Demonstrate
    1. Choose formation
      1. Standard formation
      2. With Senzing API Server formation
      3. Withinfo formation
      4. Redoer queuing formation
      5. Withinfo and Redoer formation
      6. Withinfo and Redoer queuing formation
  3. Advanced
  4. Related artifacts
  5. References

Preamble

At Senzing, we strive to create GitHub documentation in a "don't make me think" style. For the most part, instructions are copy and paste. Icons are used to signify additional actions by the user. If the instructions are not clear, please let us know by opening a new Documentation issue describing where we can improve. Now on with the show...

Expectations

  • Space: This repository and demonstration require 7 GB free disk space.
  • Time: Budget 2 hours to get the demonstration up-and-running, depending on CPU and network speeds.
  • Background knowledge: This repository assumes a working knowledge of:

Prerequisites

  1. docker - Minimum version: 20.10.16
  2. docker-compose - Minimum version: 1.29.0

Demonstrate

  1. ✏️ Specify AWS credentials. Example:

    export AWS_ACCESS_KEY_ID=$(aws configure get default.aws_access_key_id)
export AWS_SECRET_ACCESS_KEY=$(aws configure get default.aws_secret_access_key)
export AWS_DEFAULT_REGION=$(aws configure get default.region)

  2. Create AWS SQS queues at console.aws.amazon.com/sqs/home.

  3. ✏️ Specify AWS SQS queues: Example:

    export SENZING_SQS_FAILURE_QUEUE_URL="https://sqs.us-east-1.amazonaws.com/000000000000/senzing-failure-queue"
export SENZING_SQS_INFO_QUEUE_URL="https://sqs.us-east-1.amazonaws.com/000000000000/senzing-info-queue"
export SENZING_SQS_QUEUE_URL="https://sqs.us-east-1.amazonaws.com/000000000000/senzing-queue"
export SENZING_SQS_REDO_QUEUE_URL="https://sqs.us-east-1.amazonaws.com/000000000000/senzing-redo-queue"

Choose formation

🤔 Choose a docker-compose.yaml file from list. Choose one value for SENZING_DOCKER_COMPOSE_FILE from the examples given below.

Standard formation

  1. Standard demonstration.

    export SENZING_DOCKER_COMPOSE_FILE=resources/postgresql/docker-compose-sqs-postgresql.yaml

With Senzing API Server formation

Uses senzing/senzing-api-server instead of senzing/senzing-poc-server.

  1. Standard demonstration.

    export SENZING_DOCKER_COMPOSE_FILE=resources/postgresql/docker-compose-sqs-postgresql-api-server.yaml

Withinfo formation

  1. Return information with each record added to Senzing.

    export SENZING_DOCKER_COMPOSE_FILE=resources/postgresql/docker-compose-sqs-postgresql-withinfo.yaml

Redoer queuing formation

  1. Add multiple redoers to standard demonstration. This will process the Senzing "redo records". One redoer will populate sqs with redo records. One or more redoers will read redo records from sqs topic and send to the Senzing Engine.

    export SENZING_DOCKER_COMPOSE_FILE=resources/postgresql/docker-compose-sqs-postgresql-redoer-sqs.yaml

Withinfo and Redoer formation

  1. Add redoer to standard demonstration. Also, return information with each record added to Senzing.

    export SENZING_DOCKER_COMPOSE_FILE=resources/postgresql/docker-compose-sqs-postgresql-redoer-withinfo.yaml

Withinfo and Redoer queuing formation

  1. Add multiple redoers to standard demonstration. This will process the Senzing "redo records". One redoer will populate sqs with redo records. One or more redoers will read redo records from sqs topic and send to the Senzing Engine. Also, return information with each record added to Senzing.

    export SENZING_DOCKER_COMPOSE_FILE=resources/postgresql/docker-compose-sqs-postgresql-redoer-sqs-withinfo.yaml

Run formation

  1. ✏️ Specify a new directory to hold demonstration artifacts on the local host. Example:

    export SENZING_DEMO_DIR=~/my-senzing
    1. ⚠️ macOS - File sharing must be enabled for SENZING_DEMO_DIR.
    2. ⚠️ Windows - File sharing must be enabled for SENZING_DEMO_DIR.

  2. Set environment variables. Example:

    export PGADMIN_DIR=${SENZING_DEMO_DIR}/pgadmin
export POSTGRES_DIR=${SENZING_DEMO_DIR}/postgres
export SENZING_GID=$(id -g)
export SENZING_UID=$(id -u)
export SENZING_VAR_DIR=${SENZING_DEMO_DIR}/var

  3. Create directories. Example:

    mkdir -p ${PGADMIN_DIR} ${POSTGRES_DIR} ${SENZING_VAR_DIR}
chmod -R 777 ${SENZING_DEMO_DIR}

  4. Get versions of Docker images. Example:

    curl -X GET \
    --output ${SENZING_DEMO_DIR}/docker-versions-stable.sh \
    https://raw.githubusercontent.com/Senzing/knowledge-base/main/lists/docker-versions-stable.sh
source ${SENZING_DEMO_DIR}/docker-versions-stable.sh

  5. Download docker-compose.yaml and Docker images. Example:

    curl -X GET \
    --output ${SENZING_DEMO_DIR}/docker-compose.yaml \
    "https://raw.githubusercontent.com/senzing-garage/docker-compose-demo/main/${SENZING_DOCKER_COMPOSE_FILE}"
cd ${SENZING_DEMO_DIR}
sudo --preserve-env docker-compose pull

  6. Bring up Senzing docker-compose stack. Example:

    cd ${SENZING_DEMO_DIR}
sudo --preserve-env docker-compose up

  7. Allow time for the components to be downloaded, start, and initialize.

    1. There will be errors in some Docker logs as they wait for dependent services to become available. docker-compose isn't the best at orchestrating Docker container dependencies.

View formation

Once the docker-compose formation is running, different aspects of the formation can be viewed.

Username and password for the following sites were either passed in as environment variables or are the default values seen in ${SENZING_DOCKER_COMPOSE_FILE}.

View Docker containers

  1. A good tool to monitor individual Docker logs is Portainer. When running, Portainer is viewable at localhost:9170.

View Senzing Entity Search WebApp

  1. Senzing Entity Search WebApp is viewable at localhost:8251.
  2. See additional tips for working with Senzing Entity Search WebApp.

View X-Term

The web-based Senzing X-term can be used to run Senzing command-line programs.

  1. Senzing X-term is viewable at localhost:8254.
  2. See additional tips for working with Senzing X-Term.

Use SSH

Instructions to use the senzing/sshd container are viewable in the senzing/docker-sshd repository

View Jupyter notebooks

  1. Change file permissions on database files. Example:

    sudo chmod 777 -R ${POSTGRES_DIR}

  2. Jupyter Notebooks are viewable at localhost:9178.

  3. See additional tips for working with Jupyter Notebooks.

View Senzing API Server

View results from Senzing REST API server. The server supports the Senzing REST API.

  1. The OpenApi Editor with Servers value of http://localhost:8250 can be used to try the Senzing REST API.
  2. Example Senzing REST API request: localhost:8250/heartbeat
  3. See additional tips for working with Senzing API server.

View AWS SQS

  1. AWS SQS is viewable at console.aws.amazon.com/sqs/home.

View PostgreSQL

  1. PostgreSQL is viewable at localhost:9171.
    1. Database defaults: username: postgres password: postgres
  2. See additional tips for working with PostgreSQL.

Cleanup

When the docker-compose formation is no longer needed, it can be brought down and directories can be deleted.

  1. Bring down Docker formation. Example:

    cd ${SENZING_DEMO_DIR}
sudo docker-compose down

  2. Remove directories from host system. Example:

    rm -rf ${SENZING_DEMO_DIR:-/tmp/nowhere/for/safety}

Advanced

The following topics discuss variations to the basic docker-compose demonstration.

  1. SSH port
  2. Set sshd password

Docker images

This docker formation brings up the following docker containers:

  1. bitnami/postgres
  2. dpage/pgadmin4
  3. senzing/console
  4. senzing/entity-web-search-app
  5. senzing/jupyter
  6. senzing/redoer
  7. senzing/senzing-poc-server
  8. senzing/senzing-tools
  9. senzing/sshd
  10. senzing/stream-loader
  11. senzing/stream-producer
  12. senzing/xterm

Configuration

Configuration values specified by environment variable or command line parameter.

Program parameter matrix

  1. The matrix for using sqs with stream-loader.py and redoer.py subcommands.

    +-------------------------- stream-loader.py sqs
|  +----------------------- stream-loader.py sqs-withinfo
|  |  +-------------------- redoer.py redo
|  |  |  +----------------- redoer.py redo-withinfo-sqs
|  |  |  |  +-------------- redoer.py write-to-sqs
|  |  |  |  |  +----------- redoer.py read-from-sqs
|  |  |  |  |  |  +-------- redoer.py read-from-sqs-withinfo
|  |  |  |  |  |  |
v  v  v  v  v  v  v
X  .  .  .  .  .  .
X  .  X  .  .  .  .  docker-compose-sqs-postgresql.yaml
X  .  .  X  .  .  .
X  .  .  .  X  X  .  docker-compose-sqs-postgresql-redoer-sqs.yaml
X  .  .  .  X  .  X
.  X  .  .  .  .  .  docker-compose-sqs-postgresql-withinfo.yaml
.  X  X  .  .  .  .
.  X  .  X  .  .  .  docker-compose-sqs-postgresql-redoer-withinfo.yaml
.  X  .  .  X  X  .
.  X  .  .  X  .  X  docker-compose-sqs-postgresql-redoer-sqs-withinfo.yaml

Related artifacts

  1. DockerHub

References