Release 1.9.2 of the Amazon Kinesis Client

Release 1.9.2 (September 4, 2018)

• Allow use of Immutable Clients
  • Issue #280
  • PR #305
• Allow the use of AT_TIMESTAMP for MultiLang Daemon Clients.
  • [Issue #341)(https://github.com//issues/341)
  • PR #342
• Update the cache for KinesisProxy#getShard on cache misses.
  • PR #344
• Changed release process to use a standard process.
  • PR #389
• Removed tests that expected a null region response for unknown regions.
  • PR #346
• Updated the version of the AWS Java SDK to 1.11.400

Release 2.0.1 of the Amazon Kinesis Client Library for Java

Release 2.0.1 (August 21, 2018)

• Mark certain internal components with @KinesisClientInternalApi attribute.
  Components marked as internal may be deprecated at a faster rate than public components.
  • PR #358
• Fixed an issue where ResourceNotFoundException on subscription to a shard was not triggering end of shard handling.
  If a lease table contains a shard that is no longer present in the stream attempt to subscribe to that shard will trigger a ResourceNotFoundException. These exception are treated the same as reaching the end of a shard.
  • PR #359
• Fixed an issue where the KCL would not Use the configured DynamoDB IOPs when creating the lease table.
  • PR #360
• Make the maximum number of Scheduler initialization attempts configurable.
  The maximum number of Scheduler initialization attempts can be configured via CoordinatorConfig#maxInitializationAttempts.
  • PR #363
  • PR #368
• Fixed an issue where it was possible to get a duplicate record when resubscribing to a shard.
  Subscribe to shard requires periodic resubscribing, and uses a new concept of a continuation sequence number. If the continuation sequence number was equal to the last record that record would be processed a second time. Resubscribing now uses AFTER_SEQUENCE_NUMBER to ensure that only later records are returned.
  • PR #371
• Upgraded to AWS SDK 2.0.1
  • PR #372
• Fixed an issue where time based restart of the subscription wasn't resetting the lastRequestTime.
  If a subscription hasn't delivered any data for more than 30 seconds it will be canceled and restarted. This detection is based of the lastRequestTime which wasn't getting reset after the restart was triggered.
  • PR #373
• Fixed an issue where requesting on the subscription from the FanOutRecordsPublisher could trigger an unexpected failure.
  Due to a race condition the underlying flow in the subscription could be set to something else. The method is now synchronized, and verifies that the subscriber it was created with is still the subscriber in affect.
  This issue generally would only appear when multiple errors were occurring while connecting to Kinesis.
  • PR #374
• Fixed an issue where the number of requested items could exceed the capacity of the RxJava queue.
  There was an off by one issue when determining whether to make a request to the SDK subscription. This changes the calculation to represent the capacity as a queue.
  • PR #375

Release 2.0.0 of the Amazon Kinesis Client Library for Java

Release 2.0.0

• The Maven groupId, along with the version, for the Amazon Kinesis Client has changed from com.amazonaws to software.amazon.kinesis.
  To add a dependency on the new version of the Amazon Kinesis Client:

  <dependency>
      <groupId>software.amazon.kinesis</groupId>
      <artifactId>amazon-kinesis-client</artifactId>
      <version>2.0.0</version>
  </dependency>

• Added support for Enhanced Fan Out.
  Enhanced Fan Out provides for lower end to end latency, and increased number of consumers per stream.

  • Records are now delivered via streaming, reducing end-to-end latency.
  • The Amazon Kinesis Client will automatically register a new consumer if required.
    When registering a new consumer, the Kinesis Client will default to the application name unless configured otherwise.
  • SubscribeToShard maintains long lived connections with Kinesis, which in the AWS Java SDK 2.0 is limited by default.
    The KinesisClientUtil has been added to assist configuring the maxConcurrency of the KinesisAsyncClient.
    WARNING: The Amazon Kinesis Client may see significantly increased latency, unless the KinesisAsyncClient is configured to have a maxConcurrency high enough to allow all leases plus additional usages of the KinesisAsyncClient.
  • The Amazon Kinesis Client now uses 3 additional Kinesis API's:
    WARNING: If using a restrictive Kinesis IAM policy you may need to add the following API methods to the policy.
    • SubscribeToShard
    • DescribeStreamSummary
    • DescribeStreamConsumer
    • RegisterStreamConsumer
  • New configuration options are available to configure Enhanced Fan Out.

  Name	Default	Description
  consumerArn	Unset	The ARN for an already created consumer. If this is set, the Kinesis Client will not attempt to create a consumer.
  streamName	Unset	The name of the stream that a consumer should be create for if necessary
  consumerName	Unset	The name of the consumer to create. If this is not set the applicationName will be used instead.
  applicationName	Unset	The name of the application. This is used as the name of the consumer unless consumerName is set.

• Modular Configuration of the Kinesis Client
  The Kinesis Client has migrated to a modular configuration system, and the KinesisClientLibConfiguration class has been removed.
  Configuration has been split into 7 classes. Default versions of the configuration can be created from the ConfigsBuilder.
  Please [see the migration guide for more information][migration-guide].

  • CheckpointConfig
  • CoordinatorConfig
  • LeaseManagementConfig
  • LifecycleConfig
  • MetricsConfig
  • ProcessorConfig
  • RetrievalConfig

• Upgraded to AWS Java SDK 2.0
  The Kinesis Client now uses the AWS Java SDK 2.0. The dependency on AWS Java SDK 1.11 has been removed.
  All configurations will only accept 2.0 clients.

  • When configuring the KinesisAsyncClient the KinesisClientUtil#createKinesisAsyncClient can be used to configure the Kinesis Client
  • If you need support for AWS Java SDK 1.11 you will need to add a direct dependency.
    When adding a dependency you must ensure that the 1.11 versions of Jackson dependencies are excluded
    [Please see the migration guide for more information][migration-guide]

• MultiLangDaemon is now a separate module
  The MultiLangDaemon has been separated to its own Maven module and is no longer available in amazon-kinesis-client. To include the MultiLangDaemon, add a dependency on amazon-kinesis-client-multilang.

Release 1.9.1 of the Amazon Kinesis Client for Java

Release 1.9.1 (April 30, 2018)

• Added the ability to create a prepared checkpoint when at SHARD_END.
  • PR #301
• Added the ability to subscribe to worker state change events.
  • PR #291
• Added support for custom lease managers.
  A custom LeaseManager can be provided to Worker.Builder that will be used to provide lease services.
  This makes it possible to implement custom lease management systems in addition to the default DynamoDB system.
  • PR #297
• Updated the version of the AWS Java SDK to 1.11.219

Release 1.9.0 of the Amazon Kinesis Client for Java

Latest Release (1.9.0)

• Introducing support for ListShards API. This API is used in place of DescribeStream API to provide more throughput during ShardSyncTask. Please consult the AWS Documentation for ListShards for more information.

  • ListShards supoorts higher call rate, which should reduce instances of throttling when attempting to synchronize the shard list.

  • Added configuration parameters for ListShards usage

    Name	Default	Description
    listShardsBackoffTimeInMillis	1500 ms	This is the default backoff time between 2 ListShards calls when throttled.
    listShardsRetryAttempts	50	This is the maximum number of times the KinesisProxy will retry to make ListShards calls on being throttled.

• Updating the version of AWS Java SDK to 1.11.272.

  • Version 1.11.272 is now the minimum support versiojn of the SDK.

• Deprecating the following methods, and classes. These methods, and classes will be removed in a future release.

  • Deprecated IKinesisProxy#getStreamInfo.
  • Deprecated IKinesisProxyFactory.
  • Deprecated KinesisProxyFactory.
  • Deprecated certain KinesisProxy constructors.

PR #293

Release 1.8.10 of the Amazon Kinesis Client for Java

Release 1.8.10

• Allow providing a custom IKinesisProxy implementation.
  • PR #274
• Checkpointing on a different thread should no longer emit a warning about NullMetricsScope.
  • PR #284
  • Issue #48
• Upgraded the AWS Java SDK to version 1.11.271
  • PR #287

Release 1.8.9 of the Amazon Kinesis Client for Java

Release 1.8.9

• Allow disabling check for the case where a child shard has an open parent shard.
  There is a race condition where it's possible for the a parent shard to appear open, while having child shards. This check can now be disabled by setting ignoreUnexpectedChildShards to true.
  • PR #240
  • Issue #210
• Upgraded the AWS SDK for Java to 1.11.261
  • PR #281

Release 1.8.8 of the Amazon Kinesis Client for Java

Release 1.8.8

• Fixed issues with leases losses due to ExpiredIteratorException in PrefetchGetRecordsCache and AsynchronousFetchingStrategy.
  PrefetchGetRecordsCache will request for a new iterator and start fetching data again.

  • PR#263

• Added warning message for long running tasks.
  Logging long running tasks can be enabled by setting the following configuration property:

  Name	Default	Description
  logWarningForTaskAfterMillis	Not set	Milliseconds after which the logger will log a warning message for the long running task

  • PR#259

• Handling spurious lease renewal failures gracefully.
  Added better handling of DynamoDB failures when updating leases. These failures would occur when a request to DynamoDB appeared to fail, but was actually successful.

  • PR#247

• ShutdownTask gets retried if the previous attempt on the ShutdownTask fails.

  • PR#267

• Fix for using maxRecords from KinesisClientLibConfiguration in GetRecordsCache for fetching records.

  • PR#264

Release 1.8.7 of the Amazon Kinesis Client Library for Java

Release 1.8.7

• Don't add a delay for synchronous requests to Kinesis
  Removes a delay that had been added for synchronous GetRecords calls to Kinesis.
  • PR #256

Release 1.8.6 of the Amazon Kinesis Client Library for Java

Release 1.8.6

• Add prefetching of records from Kinesis
  Prefetching will retrieve and queue additional records from Kinesis while the application is processing existing records.
  Prefetching can be enabled by setting dataFetchingStrategy to PREFETCH_CACHED. Once enabled an additional fetching thread will be started to retrieve records from Kinesis. Retrieved records will be held in a queue until the application is ready to process them.
  Pre-fetching supports the following configuration values:

  Name	Default	Description
  dataFetchingStrategy	DEFAULT	Which data fetching strategy to use
  maxPendingProcessRecordsInput	3	The maximum number of process records input that can be queued
  maxCacheByteSize	8 MiB	The maximum number of bytes that can be queued
  maxRecordsCount	30,000	The maximum number of records that can be queued
  idleMillisBetweenCalls	1,500 ms	The amount of time to wait between calls to Kinesis

  • PR #246