Merge pull request #735 from ashwing/v1.x-ltr-release

Merging LTR changes to public KCL 1.x branch. Preparing for KCL 1.14.0 release
awslabs · Aug 18, 2020 · 6fbfc21 · 6fbfc21
2 parents 4f927da + ff004d7
commit 6fbfc21
Show file tree

Hide file tree

Showing 53 changed files with 5,499 additions and 1,092 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,52 @@
 # Changelog
-## Latest Release (1.13.3 March 2, 2020)
+
+## Latest Release (1.14.0 - August 17, 2020)
+
+* [Milestone#50](https://github.com/awslabs/amazon-kinesis-client/milestone/50)
+
+* Behavior of shard synchronization is moving from each worker independently learning about all existing shards to workers only discovering the children of shards that each worker owns. This optimizes memory usage, lease table IOPS usage, and number of calls made to kinesis for streams with high shard counts and/or frequent resharding.
+* When bootstrapping an empty lease table, KCL utilizes the ListShard API's filtering option (the ShardFilter optional request parameter) to retrieve and create leases only for a snapshot of shards open at the time specified by the ShardFilter parameter. The ShardFilter parameter enables you to filter out the response of the ListShards API, using the Type parameter. KCL uses the Type filter parameter and the following of its valid values to identify and return a snapshot of open shards that might require new leases.
+	* Currently, the following shard filters are supported:
+		* `AT_TRIM_HORIZON` - the response includes all the shards that were open at `TRIM_HORIZON`.
+		* `AT_LATEST` - the response includes only the currently open shards of the data stream.
+	    * `AT_TIMESTAMP` - the response includes all shards whose start timestamp is less than or equal to the given timestamp and end timestamp is greater than or equal to the given timestamp or still open.
+	* `ShardFilter` is used when creating leases for an empty lease table to initialize leases for a snapshot of shards specified at `KinesisClientLibConfiguration#initialPositionInStreamExtended`.
+	* For more information about ShardFilter, see the [official AWS documentation on ShardFilter](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_ShardFilter.html).
+
+* Introducing support for the `ChildShards` response of the `GetRecords` API to perform lease/shard synchronization that happens at `SHARD_END` for closed shards, allowing a KCL worker to only create leases for the child shards of the shard it finished processing.
+	* For KCL 1.x applications, this uses the `ChildShards` response of the `GetRecords` API.
+	* For more information, see the official AWS Documentation on [GetRecords](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html) and [ChildShard](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_ChildShard.html).
+
+* KCL now also performs additional periodic shard/lease scans in order to identify any potential holes in the lease table to ensure the complete hash range of the stream is being processed and create leases for them if required. When `KinesisClientLibConfiguration#shardSyncStrategyType` is set to `ShardSyncStrategyType.SHARD_END`, `PeriodicShardSyncManager#leasesRecoveryAuditorInconsistencyConfidenceThreshold` will be used to determine the threshold for number of consecutive scans containing holes in the lease table after which to enforce a shard sync. When `KinesisClientLibConfiguration#shardSyncStrategyType` is set to `ShardSyncStrategyType.PERIODIC`, `leasesRecoveryAuditorInconsistencyConfidenceThreshold` is ignored.
+	* New configuration options are available to configure `PeriodicShardSyncManager` in `KinesisClientLibConfiguration`
+
+	| Name                                                  | Default | Description                                                                                                                                                                                                                                                                                       |
+	| ----------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+	| leasesRecoveryAuditorInconsistencyConfidenceThreshold | 3       | Confidence threshold for the periodic auditor job to determine if leases for a stream in the lease table is inconsistent. If the auditor finds same set of inconsistencies consecutively for a stream for this many times, then it would trigger a shard sync. Only used for `ShardSyncStrategyType.SHARD_END`. |
+
+	* New CloudWatch metrics are also now emitted to monitor the health of `PeriodicShardSyncManager`:
+
+	| Name                        | Description                                            |
+	| --------------------------- | ------------------------------------------------------ |
+	| NumStreamsWithPartialLeases | Number of streams that had holes in their hash ranges. |
+	| NumStreamsToSync            | Number of streams which underwent a full shard sync.   |
+
+* Introducing deferred lease cleanup. Leases will be deleted asynchronously by `LeaseCleanupManager` upon reaching `SHARD_END`, when a shard has either expired past the stream’s retention period or been closed as the result of a resharding operation.
+	* New configuration options are available to configure `LeaseCleanupManager`.
+
+	| Name                                | Default    | Description                                                                                               |
+	| ----------------------------------- | ---------- | --------------------------------------------------------------------------------------------------------- |
+	| leaseCleanupIntervalMillis          | 1 minute   | Interval at which to run lease cleanup thread.                                                            |
+	| completedLeaseCleanupIntervalMillis | 5 minutes  | Interval at which to check if a lease is completed or not.                                                |
+	| garbageLeaseCleanupIntervalMillis   | 30 minutes | Interval at which to check if a lease is garbage (i.e trimmed past the stream's retention period) or not. |
+
+* Including an optimization to `KinesisShardSyncer` to only create leases for one layer of shards.
+* Changing default shard prioritization strategy to be `NoOpShardPrioritization` to allow prioritization of completed shards. Customers who are upgrading to this version and are reading from `TRIM_HORIZON` should continue using `ParentsFirstShardPrioritization` while upgrading.
+* Upgrading version of AWS SDK to 1.11.844.
+* [#719](https://github.com/awslabs/amazon-kinesis-client/pull/719) Upgrading version of Google Protobuf to 3.11.4.
+* [#712](https://github.com/awslabs/amazon-kinesis-client/pull/712) Allowing KCL to consider lease tables in `UPDATING` healthy.
+
+## Release 1.13.3 (1.13.3 March 2, 2020)
 [Milestone#49] (https://github.com/awslabs/amazon-kinesis-client/milestone/49）
 * Refactoring shard closure verification performed by ShutdownTask.
   * [PR #684] (https://github.com/awslabs/amazon-kinesis-client/pull/684)

diff --git a/README.md b/README.md
@@ -31,28 +31,51 @@ To make it easier for developers to write record processors in other languages,
 
 ## Release Notes
 
-#### Latest Release (1.13.3 March 2, 2020)
-* Refactoring shard closure verification performed by ShutdownTask.
-  * [PR #684] (https://github.com/awslabs/amazon-kinesis-client/pull/684)
-* Fixing the bug in ShardSyncTaskManager to resolve the issue of new shards not being processed after resharding.
-  * [PR #694] (https://github.com/awslabs/amazon-kinesis-client/pull/694)
-
-#### Release (1.13.2 Janurary 13, 2020)
-* Adding backward compatible constructors that use the default DDB Billing Mode (#673)
-  * [PR #673](https://github.com/awslabs/amazon-kinesis-client/pull/673)
-
-#### Release (1.13.1 December 30, 2019)
-* Adding BillingMode Support to KCL 1.x. This enables the customer to specify if they want provisioned capacity for DDB, or pay per request.
-  * [PR #656](https://github.com/awslabs/amazon-kinesis-client/pull/656)
-* Ensure ShardSyncTask invocation from ShardSyncTaskManager for pending ShardEnd events.
-  * [PR #659](https://github.com/awslabs/amazon-kinesis-client/pull/659)
-* Fix the LeaseManagementIntegrationTest failure.
-  * [PR #670](https://github.com/awslabs/amazon-kinesis-client/pull/670)
-
-#### Release (1.13.0 November 5, 2019)
-* Handling completed and blocked tasks better during graceful shutdown
-  * [PR #640](https://github.com/awslabs/amazon-kinesis-client/pull/640)
-
+### Latest Release (1.14.0 - August 17, 2020)
+
+* [Milestone#50](https://github.com/awslabs/amazon-kinesis-client/milestone/50)
+
+* Behavior of shard synchronization is moving from each worker independently learning about all existing shards to workers only discovering the children of shards that each worker owns. This optimizes memory usage, lease table IOPS usage, and number of calls made to kinesis for streams with high shard counts and/or frequent resharding.
+* When bootstrapping an empty lease table, KCL utilizes the ListShard API's filtering option (the ShardFilter optional request parameter) to retrieve and create leases only for a snapshot of shards open at the time specified by the ShardFilter parameter. The ShardFilter parameter enables you to filter out the response of the ListShards API, using the Type parameter. KCL uses the Type filter parameter and the following of its valid values to identify and return a snapshot of open shards that might require new leases.
+	* Currently, the following shard filters are supported:
+		* `AT_TRIM_HORIZON` - the response includes all the shards that were open at `TRIM_HORIZON`.
+		* `AT_LATEST` - the response includes only the currently open shards of the data stream.
+	    * `AT_TIMESTAMP` - the response includes all shards whose start timestamp is less than or equal to the given timestamp and end timestamp is greater than or equal to the given timestamp or still open.
+	* `ShardFilter` is used when creating leases for an empty lease table to initialize leases for a snapshot of shards specified at `KinesisClientLibConfiguration#initialPositionInStreamExtended`.
+	* For more information about ShardFilter, see the [official AWS documentation on ShardFilter](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_ShardFilter.html).
+
+* Introducing support for the `ChildShards` response of the `GetRecords` API to perform lease/shard synchronization that happens at `SHARD_END` for closed shards, allowing a KCL worker to only create leases for the child shards of the shard it finished processing.
+	* For KCL 1.x applications, this uses the `ChildShards` response of the `GetRecords` API.
+	* For more information, see the official AWS Documentation on [GetRecords](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html) and [ChildShard](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_ChildShard.html).
+
+* KCL now also performs additional periodic shard/lease scans in order to identify any potential holes in the lease table to ensure the complete hash range of the stream is being processed and create leases for them if required. When `KinesisClientLibConfiguration#shardSyncStrategyType` is set to `ShardSyncStrategyType.SHARD_END`, `PeriodicShardSyncManager#leasesRecoveryAuditorInconsistencyConfidenceThreshold` will be used to determine the threshold for number of consecutive scans containing holes in the lease table after which to enforce a shard sync. When `KinesisClientLibConfiguration#shardSyncStrategyType` is set to `ShardSyncStrategyType.PERIODIC`, `leasesRecoveryAuditorInconsistencyConfidenceThreshold` is ignored.
+	* New configuration options are available to configure `PeriodicShardSyncManager` in `KinesisClientLibConfiguration`
+
+	| Name                                                  | Default | Description                                                                                                                                                                                                                                                                                       |
+	| ----------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+	| leasesRecoveryAuditorInconsistencyConfidenceThreshold | 3       | Confidence threshold for the periodic auditor job to determine if leases for a stream in the lease table is inconsistent. If the auditor finds same set of inconsistencies consecutively for a stream for this many times, then it would trigger a shard sync. Only used for `ShardSyncStrategyType.SHARD_END`. |
+
+	* New CloudWatch metrics are also now emitted to monitor the health of `PeriodicShardSyncManager`:
+
+	| Name                        | Description                                            |
+	| --------------------------- | ------------------------------------------------------ |
+	| NumStreamsWithPartialLeases | Number of streams that had holes in their hash ranges. |
+	| NumStreamsToSync            | Number of streams which underwent a full shard sync.   |
+
+* Introducing deferred lease cleanup. Leases will be deleted asynchronously by `LeaseCleanupManager` upon reaching `SHARD_END`, when a shard has either expired past the stream’s retention period or been closed as the result of a resharding operation.
+	* New configuration options are available to configure `LeaseCleanupManager`.
+
+	| Name                                | Default    | Description                                                                                               |
+	| ----------------------------------- | ---------- | --------------------------------------------------------------------------------------------------------- |
+	| leaseCleanupIntervalMillis          | 1 minute   | Interval at which to run lease cleanup thread.                                                            |
+	| completedLeaseCleanupIntervalMillis | 5 minutes  | Interval at which to check if a lease is completed or not.                                                |
+	| garbageLeaseCleanupIntervalMillis   | 30 minutes | Interval at which to check if a lease is garbage (i.e trimmed past the stream's retention period) or not. |
+
+* Including an optimization to `KinesisShardSyncer` to only create leases for one layer of shards.
+* Changing default shard prioritization strategy to be `NoOpShardPrioritization` to allow prioritization of completed shards. Customers who are upgrading to this version and are reading from `TRIM_HORIZON` should continue using `ParentsFirstShardPrioritization` while upgrading.
+* Upgrading version of AWS SDK to 1.11.844.
+* [#719](https://github.com/awslabs/amazon-kinesis-client/pull/719) Upgrading version of Google Protobuf to 3.11.4.
+* [#712](https://github.com/awslabs/amazon-kinesis-client/pull/712) Allowing KCL to consider lease tables in `UPDATING` healthy.
 
 ###### For remaining release notes check **[CHANGELOG.md][changelog-md]**.
 

diff --git a/pom.xml b/pom.xml
@@ -6,7 +6,7 @@
   <artifactId>amazon-kinesis-client</artifactId>
   <packaging>jar</packaging>
   <name>Amazon Kinesis Client Library for Java</name>
-  <version>1.13.4-SNAPSHOT</version>
+  <version>1.14.0</version>
   <description>The Amazon Kinesis Client Library for Java enables Java developers to easily consume and process data
     from Amazon Kinesis.
   </description>
@@ -25,7 +25,7 @@
   </licenses>
 
   <properties>
-    <aws-java-sdk.version>1.11.728</aws-java-sdk.version>
+    <aws-java-sdk.version>1.11.844</aws-java-sdk.version>
     <sqlite4java.version>1.0.392</sqlite4java.version>
     <sqlite4java.native>libsqlite4java</sqlite4java.native>
     <sqlite4java.libpath>${project.build.directory}/test-lib</sqlite4java.libpath>

diff --git a/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ConsumerStates.java b/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ConsumerStates.java
@@ -199,7 +199,7 @@ public ConsumerState successTransition() {
 
         @Override
         public ConsumerState shutdownTransition(ShutdownReason shutdownReason) {
-            return ShardConsumerState.SHUTDOWN_COMPLETE.getConsumerState();
+            return ShardConsumerState.SHUTTING_DOWN.getConsumerState();
         }
 
         @Override
@@ -530,7 +530,9 @@ public ITask createTask(ShardConsumer consumer) {
                     consumer.isIgnoreUnexpectedChildShards(),
                     consumer.getLeaseCoordinator(),
                     consumer.getTaskBackoffTimeMillis(),
-                    consumer.getGetRecordsCache(), consumer.getShardSyncer(), consumer.getShardSyncStrategy());
+                    consumer.getGetRecordsCache(), consumer.getShardSyncer(),
+                    consumer.getShardSyncStrategy(), consumer.getChildShards(),
+                    consumer.getLeaseCleanupManager());
         }
 
         @Override