Carabas: Kinesis, DynamoDB, DMS - CDC record processing in different flavours #120
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
amotl
force-pushed
the
kinesis
branch
6 times, most recently
from
9533b64 to
eba58e0
Compare
amotl
changed the title
amotl
force-pushed
the
kinesis
branch
6 times, most recently
from
6c255ac to
ff69b68
Compare
amotl
commented
Jul 13, 2024
amotl
commented
Jul 25, 2024
There was a problem hiding this comment.
This could also be interesting in this context.
Lambda Forge is a Python framework that simplifies AWS Lambda function creation and deployment. It automates serverless management, letting developers focus on coding.
amotl
force-pushed
the
kinesis
branch
4 times, most recently
from
3ffae15 to
12a7c3e
Compare
amotl
force-pushed
the
kinesis
branch
2 times, most recently
from
9b3bfbc to
ad33a3f
Compare
amotl
changed the title
amotl
force-pushed
the
kinesis
branch
3 times, most recently
from
90dabdc to
ff13238
Compare
It needs further curation. The next iteration should aim towards slotting it in as a native streamz Source element.
- Software Tests - Configuration - Documentation - Cleanups - Fixes
- Batch Size: 2500 - Memory Size: 512 MB
The previous one didn't include support for DMS Serverless.
- Configure ReplicationType to use `full-load-and-cdc`. - Configure ReplicationSettings to use `EnableBeforeImage`. - Add RDSParameterGroup to configure pgaudit, pglogical, and pg_stat_statements plugins. - Configure DMS source endpoint (PostgreSQL) to use pglogical. - Configure DMS target endpoint (Kinesis) to include all optional details: ControlDetails, PartitionValue, TransactionDetails, NullAndEmpty, TableAlterOperations, IncludeSchemaTable - Add `RDSInstanceArn` output variable. - Add `ReplicationArn` output variable.
Dependency woes about `requests`, `docker`, and `pytest-asyncio-cooperative`?
Because synchronous and asynchronous tests are mixed, and maybe because of woes with pytest fixtures, the test suite must turn off concurrency.
- Use a real DMS replication instance - Streamline configuration of DMS replication task - Improve processor Lambda
- Provide new IaC entrypoint options `batch_size`, `starting_position`, and `starting_position_timestamp`. - Add software integration test case for Kinesis/DMS/CrateDB.
amotl
force-pushed
the
kinesis
branch
2 times, most recently
from
ec4fb49 to
7e0a630
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
About
Optionally pull in infrastructure managed by others.
Status
This is a work in progress.
What's Inside
About configuration and usage.
DynamoDB CDC to CrateDB using DynamoDB Streams Kinesis Adapter/cc @hammerhead, @hlcianfagna, @wierdvanderhaar, @surister