Helper classes for the AWS v3 SDK for JS Kinesis Client (@aws-sdk/client-kinesis
).
KinesisRetrier
will retry failed records with a "successful" PutRecordsCommand
that has items marked as failed within the response, which then need to be retried.
KinesisBackgroundWriter
will accept writes immediately if there is a concurrent writer slot available, and will block the caller until a slot becomes available if all slots are in use.
The package is available on npm as @shutterstock/kinesis-helpers
npm i @shutterstock/kinesis-helpers
import { KinesisBackgroundWriter, KinesisRetrier } from '@shutterstock/kinesis-helpers';
After installing the package, you might want to look at our API Documentation to learn about all the features available.
- KinesisRetrier
- Kinesis batch put retrier
- Batch puts return a list of items that were throttled
- The batch is marked as succeeded (200 status code) even though items failed
- As a result, the built-in AWS SDK retry logic will not retry the items in the batch that were throttle
- KinesisBackgroundWriter
- Accepts writes immediately if there is a concurrent writer slot available
- Blocks the caller until a slot becomes available if all slots are in use
- Collects and exposes errors so the caller can detect if errors have been happening
nvm use
npm i
npm run build
npm run lint
npm run test
- Create Kinesis Data Stream using AWS Console or any other method
- Example:
aws kinesis create-stream --stream-name kinesis-helpers-test-stream --shard-count 1
- Default name is
kinesis-helpers-test-stream
- 1 shard is sufficient
- 1 day retention is sufficient
- No encryption is sufficient
- On-demand throughput is sufficient
- Example:
npm run example:kinesis-retrier
- If the stream name was changed:
KINESIS_STREAM_NAME=my-stream-name npm run example:kinesis-retrier
- If the stream name was changed:
- Create Kinesis Data Stream using AWS Console or any other method
- Example:
aws kinesis create-stream --stream-name kinesis-helpers-test-stream --shard-count 1
- Default name is
kinesis-helpers-test-stream
- 1 shard is sufficient
- 1 day retention is sufficient
- No encryption is sufficient
- On-demand throughput is sufficient
- Example:
npm run example:kinesis-background-writer
- If the stream name was changed:
KINESIS_STREAM_NAME=my-stream-name npm run example:kinesis-background-writer
- If the stream name was changed: