Skip to content

Release 0.14.0 of the Amazon Kinesis Producer Library
Compare
Choose a tag to compare
@Cory-Bradshaw Cory-Bradshaw released this 18 Nov 22:28
· 237 commits to master since this release
v0.14.0
7753dfc
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

Required KPL Update – v0.14.0

KPL 0.14.0 now uses ListShards API, making it easier for your Kinesis Producer applications to scale. Kinesis Data Streams (KDS) enables you to scale your stream capacity without any changes to producers and consumers. After a scaling event, producer applications need to discover the new shard map. Version 0.14.0 replaces the DescribeStream with the ListShards API for shard discovery. ListShards API supports 100TPS per stream compared to DescribeStream that supports 10TPS per account z. For an account with 10 streams using KPL v0.14.0 will provide you a 100X higher call rate for shard discovery, eliminating the need for a DescribeStream API limit increase for scaling. You can find more information on the ListShards API in the Kinesis Data Streams documentation.

Recommended Settings for Streams larger than 800 shards

The KPL is an application for ingesting data to your Kinesis Data Streams. As your streams grow you may find the need to tune the KPL unable to accommodate the growing needs of your applications. Without optimized configurations your KPL processes will see inefficient CPU usage and delays in writing records into KDS. For streams larger than 800 shards, we recommend the following settings:

  • ThreadingModel= “POOLED”
  • MetricsGranularity= “stream”
  • ThreadPoolSize=128

We recommend performing sufficient testing before applying these changes to production, as every customer has different usage patterns

Release Notes

0.14.0

  • Note: Windows platform will be unsupported going forward for this library.
  • [PR #280] When aggregation is enabled and all the buffer time is consumed for aggregating User records into Kinesis records, allow some additional buffer time for aggregating Kinesis Records into PutRecords calls.
  • [PR #260] Added endpoint for China Ningxia region (cn-northwest-1).
  • [PR #277] Changed mechanism to update the shard map
    • Switched to using ListShards instead of DescribeStream, as this is a more scalable API
    • Reduced the number of unnecessary shard map invalidations
    • Reduced the number of unnecessary update shard map calls
    • Reduced logging noise for aggregated records landing on an unexpected shard
  • [PR #276] Updated AWS SDK from 1.0.5 to 1.7.180
  • [PR #275] Improved the sample code to avoid need to edit code to run.
  • [PR #274] Updated bootstrap.sh to build all dependencies and pack binaries into the jar.
  • [PR #273] Added compile flags to enable compiling aws-sdk-cpp with Gcc7.
  • [PR #229] Fixed bootstrap.sh to download dependent libraries directly from source.
  • [PR #246] [PR #264] Various Typos
Assets 5
Loading