v4.0.0

Release 4.0.0 (November 6, 2024)

• New lease assignment / load balancing algorithm
  • KCL 3.x introduces a new lease assignment and load balancing algorithm. It assigns leases among workers based on worker utilization metrics and throughput on each lease, replacing the previous lease count-based lease assignment algorithm.
  • When KCL detects higher variance in CPU utilization among workers, it proactively reassigns leases from over-utilized workers to under-utilized workers for even load balancing. This ensures even CPU utilization across workers and removes the need to over-provision the stream processing compute hosts.
• Optimized DynamoDB RCU usage
  • KCL 3.x optimizes DynamoDB read capacity unit (RCU) usage on the lease table by implementing a global secondary index with leaseOwner as the partition key. This index mirrors the leaseKey attribute from the base lease table, allowing workers to efficiently discover their assigned leases by querying the index instead of scanning the entire table.
  • This approach significantly reduces read operations compared to earlier KCL versions, where workers performed full table scans, resulting in higher RCU consumption.
• Graceful lease handoff
  • KCL 3.x introduces a feature called "graceful lease handoff" to minimize data reprocessing during lease reassignments. Graceful lease handoff allows the current worker to complete checkpointing of processed records before transferring the lease to another worker. For graceful lease handoff, you should implement checkpointing logic within the existing shutdownRequested() method.
  • This feature is enabled by default in KCL 3.x, but you can turn off this feature by adjusting the configuration property isGracefulLeaseHandoffEnabled.
  • While this approach significantly reduces the probability of data reprocessing during lease transfers, it doesn't completely eliminate the possibility. To maintain data integrity and consistency, it's crucial to design your downstream consumer applications to be idempotent. This ensures that the application can handle potential duplicate record processing without adverse effects.
• New DynamoDB metadata management artifacts
  • KCL 3.x introduces two new DynamoDB tables for improved lease management:
    • Worker metrics table: Records CPU utilization metrics from each worker. KCL uses these metrics for optimal lease assignments, balancing resource utilization across workers. If CPU utilization metric is not available, KCL assigns leases to balance the total sum of shard throughput per worker instead.
    • Coordinator state table: Stores internal state information for workers. Used to coordinate in-place migration from KCL 2.x to KCL 3.x and leader election among workers.
  • Follow this documentation to add required IAM permissions for your KCL application.
• Other improvements and changes
  • Dependency on the AWS SDK for Java 1.x has been fully removed.
    • The Glue Schema Registry integration functionality no longer depends on AWS SDK for Java 1.x. Previously, it required this as a transient dependency.
    • Multilangdaemon has been upgraded to use AWS SDK for Java 2.x. It no longer depends on AWS SDK for Java 1.x.
  • idleTimeBetweenReadsInMillis (PollingConfig) now has a minimum default value of 200.
    • This polling configuration property determines the publishers wait time between GetRecords calls in both success and failure cases. Previously, setting this value below 200 caused unnecessary throttling. This is because Amazon Kinesis Data Streams supports up to five read transactions per second per shard for shared-throughput consumers.
  • Shard lifecycle management is improved to deal with edge cases around shard splits and merges to ensure records continue being processed as expected.
• Migration
  • The programming interfaces of KCL 3.x remain identical with KCL 2.x for an easier migration. For detailed migration instructions, please refer to the Migrate consumers from KCL 2.x to KCL 3.x page in the Amazon Kinesis Data Streams developer guide.
• Configuration properties
  • New configuration properties introduced in KCL 3.x are listed in this doc.
  • Deprecated configuration properties in KCL 3.x are listed in this doc. You need to keep the deprecated configuration properties during the migration from any previous KCL version to KCL 3.x.
• Metrics
  • New CloudWatch metrics introduced in KCL 3.x are explained in the Monitor the Kinesis Client Library with Amazon CloudWatch in the Amazon Kinesis Data Streams developer guide. The following operations are newly added in KCL 3.x:
    • LeaseAssignmentManager
    • WorkerMetricStatsReporter
    • LeaseDiscovery

Release 3.0.1 of the Amazon Kinesis Client for .NET

Release 3.0.1 (April 24, 2024)

• Upgrade amazon-kinesis-client from 2.4.4 to 2.5.8
• Upgrade netcoreapp from 5.0 to 6.0
• Upgrade awssdk.version from 2.19.2 to 2.19.16
• Upgrade fasterxml-jackson.version from 2.13.4 to 2.14.5
• Upgrade protobuf-java from 3.21.5 to 3.23.0
• Upgrade aws-java-sdk-core from 1.12.370 to 1.12.468
• Upgrade error_prone_annotations from 2.7.1 to 2.19.1
• Upgrade NSubstitute from 3.1.0 to 5.1.0
• Upgrade Stateless from 2.5.11.0 to 5.13.0
• Upgrade MSTest.TestFramework from 1.3.2 to 3.1.1
• Upgrade MSTest.TestAdapter from 1.3.2 to 3.1.1
• Upgrade com.google.protobuf:protobuf-java from 3.21.5 to 3.21.7
• Upgrade jackson-databind from 2.13.4 to 2.13.4.2
• Upgrade guava from 31.0.1-jre to 32.1.1-jre
• Upgrade io.netty:netty-codec-http from 4.1.86.Final to 4.1.108.Final
• Upgrade Microsoft.Extensions.DependencyInjection from 6.0.0 to 8.0.0
• Upgrade ch.qos.logback:logback-core from 1.3.0 to 1.3.12
• Upgrade Microsoft.NET.Test.Sdk from 15.5.0 to 17.9.0
• Upgrade AWSSDK.Kinesis from 3.7.1.5 to 3.7.301.73
• Upgrade commons-collections4 from 4.2 to 4.4
• Upgrade commons-logging from 1.1.3 to 1.2
• Upgrade NUnit3TestAdapter from 3.9.0 to 4.5.0
• Upgrade CommandLineParser from 2.2.1 to 2.9.1
• Upgrade Microsoft.NET.Test.Sdk from 15.5.0 to 17.9.0
• Upgrade NUnit from 3.9.0 to 4.1.0
• Upgrade ion-java from 1.5.1 to 1.11.4

Full Changelog: v3.0.0...v3.0.1

Release 3.0.0 of the Amazon Kinesis Client for .NET

What's Changed

• Relicense to Apache-2.0 by @jpeddicord in #35
• Configure dependabot + Upgrade netcoreapp 5.0 by @zengyu714 in #42
• Bump fasterxml-jackson.version from 2.13.1 to 2.13.3 by @dependabot in #72
• Bump netty.version from 4.1.72.Final to 4.1.78.Final by @dependabot in #81
• Release v3.0.0 of the Amazon Kinesis Client for .Net by @zengyu714 in #119

Full Changelog: v2.0.0...v3.0.0

Release 2.0.0 of the Amazon Kinesis Client for .NET

• Added support for Enhanced Fan-Out.
  Enhanced Fan-Out provides dedicated throughput per stream consumer, and uses an HTTP/2 push API (SubscribeToShard) to deliver records with lower latency.
• Updated the Amazon Kinesis Client Library for Java to version 2.1.2.
  • Version 2.1.2 uses 4 additional Kinesis API's
    WARNING: These additional API's may require updating any explicit IAM policies
    • RegisterStreamConsumer
    • SubscribeToShard
    • DescribeStreamConsumer
    • DescribeStreamSummary
  • For more information about Enhanced Fan-Out with the Amazon Kinesis Client Library please see the announcement and developer documentation.
• Added a new record processor interface IShardRecordProcessor. This interface closely matches the Java ShardRecordProcessor interface.
  While the original IRecordProcessor interface remains present, and will continue to work it's recommended to upgrade to the newer interface.
  • The Shutdown method from IRecordProcessor has been replaced by LeaseLost and ShardEnded.
  • Added the LeaseLost method which is invoked when a lease is lost.
    LeaseLost replaces Shutdown where ShutdownInput.Reason was ShutdownReason.ZOMBIE.
  • Added the ShardEnded method which is invoked when all records from a split or merge have been processed.
    ShardEnded replaces Shutdown where ShutdownInput.Reason was ShutdownReason.TERMINATE.
  • Added ShutdownRequested which provides the record processor a last chance to checkpoint during the Amazon Kinesis Client Library shutdown process before the lease is canceled.
    • To control how long the Amazon Kinesis Client Library waits for the record processors to complete shutdown, add timeoutInSeconds=<seconds to wait> to your properties file.
• Updated the AWS Java SDK version to 2.4.0
• MultiLangDaemon now provides logging using Logback.
  • MultiLangDaemon supports custom configurations for logging via a Logback XML configuration file.
  • The Bootstrap program was been updated to accept either -l or --log-configuration to provide a Logback XML configuration file.