Changestream sources object-level change events from a MySQL Replication Master (configured for row-based replication) by streaming the MySQL binlog and transforming change events into JSON strings that can be published anywhere.
Currently, Amazon Simple Queuing Service (SQS), Amazon Simple Notification Service and Amazon S3 are supported with optional client-side message encryption via AES.
- Why SNS+SQS?
- How Do I Consume My Events?
- The Stack
- Architecture
- Binlog Event Routing
- IntelliJ
- Configuration
First, start MySQL with
docker-compose up db
(or, for the hard way without Docker, follow the bespoke setup instructions).
You will then have access to a MySQL instance on localhost port 3306 with a root password of "password", and Changestream configured to listen for events.
Next run the dev environment
docker-compose run dev
Then build and run Changestream using the sbt CLI:
$ sbt run
Note: The Dockerfile will be written to target/docker/Dockerfile
, and the docker image will be added to your local docker repo.
$ sbt docker
$ sbt debian:packageBin
$ sbt package
- Scala 2.11
- Akka 2.5
- SBT 0.13
- MySQL 5.7 (5.5+ supported)
In order to run tests locally you need to configure test.conf
with a working MySQL connection configuration.
Make sure that your test user has permissions to create and drop databases and tables. By default, Changestream uses the changestream_test
table for tests.
To run the integration tests, as well as the unit tests on the SqsActor
and SnsActor
, you also need to ensure that you have configured your AWS tokens to be available from the environment, and that your AWS tokens have access to create and add messages to queues (SQS) and create and publish to topics (SNS).
AWS_ACCESS_KEY_ID=<access_key_id>
AWS_SECRET_ACCESS_KEY=<secret_access_key>
AWS_REGION=<your_region>
AWS_S3_BUCKET=<your_bucket>
AWS_S3_KEY_PREFIX=<some_key_prefix>
Once that's all set up, you can either run
$ sbt test
from the command line, or run the test command in IntelliJ (with the Scala and SBT plugins installed).
Changestream uses ScalaStyle for code linting--it is not automatic. Before submitting a PR, please take a look at the output of the sbt scalastyle
command and correct any issues you find.
- Peter Fry
- Donovan Hubbard
- Cesar Palafox
- Jack Senechal
- Arthur Le Maitre
- Ada Ebling
- Shirish Pampoorickal
- Jon Bardin
- Saleh Rastani
- Cassandra Cruz
Changestream was inspired in a large part by mardambey/mypipe by Hisham Mardam-Bey. The first proof-of-concept for changestream was a fork of mypipe - thank you Hisham.