Skip to content

Commit

Permalink
Carabas: Update configuration
Browse files Browse the repository at this point in the history
  • Loading branch information
Andreas Motl committed Aug 16, 2024
1 parent 8bf1869 commit 20137dd
Show file tree
Hide file tree
Showing 4 changed files with 31 additions and 16 deletions.
2 changes: 1 addition & 1 deletion doc/carabas/dms/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ help.
### Install
Install LorryStream.
```shell
pip install lorrystream
pip install --upgrade 'lorrystream[carabas]'
```
Acquire IaC driver program.
```shell
Expand Down
7 changes: 4 additions & 3 deletions doc/carabas/kcl/dynamodb.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,15 +121,16 @@ aws kinesis delete-stream --stream-name dynamodb-cdc --enforce-consumer-deletion

Acquire sources and initialize sandbox.
```shell
git clone https://github.com/daq-tools/lorrystream --branch=kinesis
git clone https://github.com/daq-tools/lorrystream
cd lorrystream
python3 -m venv .venv
source .venv/bin/activate
pip install --editable='.[carabas]'
```

Install dependencies, mainly the [amazon-kclpy] package.
```shell
cd lorrystream/dynamodb_cloud
cd lorrystream/spike/kcl_dynamodb
pip install wheel
pip install --verbose -r requirements.txt
```
Expand All @@ -141,7 +142,7 @@ virtualenv on the top-level directory. Then, navigate to the playground
directory, and seed AWS credentials.
```shell
source .venv/bin/activate
cd lorrystream/dynamodb_cloud
cd lorrystream/spike/kcl_dynamodb
export AWS_ACCESS_KEY=...
export AWS_SECRET_ACCESS_KEY=...
```
Expand Down
7 changes: 4 additions & 3 deletions doc/carabas/kcl/kinesis.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,15 +34,16 @@ permissions to do so.

Acquire sources and initialize sandbox.
```shell
git clone https://github.com/daq-tools/lorrystream --branch=kinesis
git clone https://github.com/daq-tools/lorrystream
cd lorrystream
python3 -m venv .venv
source .venv/bin/activate
pip install --editable='.[carabas]'
```

Install dependencies, mainly the [amazon-kclpy] package.
```shell
cd lorrystream/kinesis
cd lorrystream/spike/kcl_kinesis
pip install wheel
pip install --verbose -r requirements.txt
```
Expand All @@ -65,7 +66,7 @@ virtualenv on the top-level directory. Then, navigate to the playground
directory, and seed AWS credentials.
```shell
source .venv/bin/activate
cd lorrystream/kinesis
cd lorrystream/spike/kcl_kinesis
export AWS_ACCESS_KEY=...
export AWS_SECRET_ACCESS_KEY=...
```
Expand Down
31 changes: 22 additions & 9 deletions doc/source/kinesis.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,19 @@
# Kinesis Source

## LocalStack Testbed
The recipe uses the LocalStack AWS environment to run an Amazon Kinesis surrogate.
This recipe uses the LocalStack AWS environment to run an Amazon Kinesis surrogate.
The walkthrough follows the [Get started with Kinesis on LocalStack] tutorial.

If you intend to invoke the commands on a real AWS environment, just use `aws`
instead of `awslocal`.

:::{tip}
LocalStack is a cloud service emulator that runs in a single container on your
laptop or in your CI environment. With LocalStack, you can run your AWS
applications or Lambdas entirely on your local machine without connecting to
a remote cloud provider.
:::

## Setup
Start the LocalStack service using Docker.
```shell
docker run \
Expand All @@ -13,17 +23,13 @@ docker run \
-v /var/run/docker.sock:/var/run/docker.sock \
localstack/localstack:3.6
```
:::{tip}
LocalStack is a cloud service emulator that runs in a single container on your
laptop or in your CI environment. With LocalStack, you can run your AWS
applications or Lambdas entirely on your local machine without connecting to
a remote cloud provider.
:::

Install LorryStream including LocalStack CLI programs.
```shell
pip install lorrystream
pip install --upgrade 'lorrystream[carabas]'
```

## Configure
Create a Kinesis Data Stream called `testdrive`.
```shell
awslocal kinesis create-stream \
Expand All @@ -43,6 +49,7 @@ Display Stream ARN.
awslocal kinesis describe-stream --stream-name testdrive | jq -r .StreamDescription.StreamARN
```

## Usage
Submit an item to the data stream, using `awslocal`.
```shell
awslocal kinesis put-record \
Expand All @@ -64,4 +71,10 @@ export AWS_ENDPOINT_URL="http://localhost:4566"
python examples/aws/kinesis_subscribe.py testdrive
```

:::{todo}
Demonstrate how to add a processor pipeline element using both either
AWS Lambda, or a dedicated processor instance.
:::


[Get started with Kinesis on LocalStack]: https://docs.localstack.cloud/user-guide/aws/kinesis/

0 comments on commit 20137dd

Please sign in to comment.