From f7e5f3ae6001eb2f43ea285a14bc1d1f8a4e05cb Mon Sep 17 00:00:00 2001 From: Greg Furman Date: Thu, 5 Jun 2025 14:07:01 +0200 Subject: [PATCH 1/5] add(kinesis): Include new Scala engine and performance section --- content/en/user-guide/aws/kinesis/index.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/content/en/user-guide/aws/kinesis/index.md b/content/en/user-guide/aws/kinesis/index.md index 7993cd9498..3eade63897 100644 --- a/content/en/user-guide/aws/kinesis/index.md +++ b/content/en/user-guide/aws/kinesis/index.md @@ -173,11 +173,27 @@ You can fetch the CloudWatch logs for your Lambda function reading records from | Variable | Description | | -------- | ----------- | +| `KINESIS_MOCK_PROVIDER_ENGINE` | String value of `node` (default) or `scala` that determines the underlying build of Kinesis Mock. [^scala-future] | +| `KINESIS_MOCK_MAXIMUM_HEAP_SIZE` | JVM memory format string (default: `512m`) that sets the maximum memory size for the Kinesis Mock Scala server, corresponds to the JVM `-Xmx` flag. [^scala] | +| `KINESIS_MOCK_INITIAL_HEAP_SIZE` | JVM memory format string (default: `256m`) that sets the initial memory size for the Kinesis Mock Scala server, corresponds to the JVM `-Xms` flag. [^scala] | | `KINESIS_ERROR_PROBABILITY` | Decimal value between `0.0` (default) and `1.0`. This environment variable enables you to inject `ProvisionedThroughputException` at random intervals in your application. While this won't provide insight into your application's overall throughput handling, it aids in testing your application's exception-handling capabilities. | | `KINESIS_SHARD_LIMIT` | Integer value (default: `100`) or `Infinity` (to disable). Use this variable to assess whether your application conforms to the assigned shard limit. Disabling this behavior requires explicitly setting `KINESIS_SHARD_LIMIT=Infinity`. | | `KINESIS_LATENCY` | Integer value in milliseconds (default: `500`) or `0` (to disable). Particularly useful for testing latency-sensitive applications. Since local Kinesis service lacks latency simulation, you can introduce artificial latency into your AWS calls using this variable. To disable this behavior, set `KINESIS_LATENCY=0`. | | `KINESIS_INITIALIZE_STREAMS` | **Deprecated.** Comma-delimited string with stream names, corresponding shard counts, and an optional region for initialization during startup. If no region is provided, the default region is used. For example, `KINESIS_INITIALIZE_STREAMS=my-first-stream:1,my-other-stream:2:us-west-2,my-last-stream:1`. In multi-account setups, the specified streams will be created for all accounts. | +### Performance Tuning + +For high-volume workloads or large payloads, we recommend switching to the Scala engine via the `KINESIS_MOCK_PROVIDER_ENGINE=scala` flag, delivering up to 10x better performance compared to the default Node.js engine. + +Additionally, the following parameters can be tuned: + +- Increase `KINESIS_MOCK_MAXIMUM_HEAP_SIZE` beyond the default `512m` to reduce JVM memory pressure. [^scala] +- Increase `KINESIS_MOCK_INITIAL_HEAP_SIZE` beyond the default `256m` to pre-allocate more JVM heap memory. [^scala] +- Reduce `KINESIS_LATENCY` artificial response delays from the default `500` milliseconds (or disable entirely with `0`). + +[^scala]: This feature is only available in the more performant Scala Kinesis Mock build that can be toggled by setting `KINESIS_MOCK_PROVIDER_ENGINE=scala`. +[^scala-future]: Future versions of LocalStack will likely default to using the `scala` engine over the less-performant `node` version currently in use. + ## Resource Browser The LocalStack Web Application provides a Resource Browser for managing Kinesis Streams & Kafka Clusters. From 7e0fc893e9c1e1978ea668a82b85f90ad4174981 Mon Sep 17 00:00:00 2001 From: HarshCasper Date: Thu, 5 Jun 2025 06:55:44 -0700 Subject: [PATCH 2/5] some changes --- content/en/references/configuration.md | 3 +++ content/en/user-guide/aws/kinesis/index.md | 22 ++++++---------------- 2 files changed, 9 insertions(+), 16 deletions(-) diff --git a/content/en/references/configuration.md b/content/en/references/configuration.md index 2656f40f5a..86d8f6c908 100644 --- a/content/en/references/configuration.md +++ b/content/en/references/configuration.md @@ -220,6 +220,9 @@ Also see [OpenSearch configuration variables]({{< ref "configuration#opensearch" | `KINESIS_SHARD_LIMIT` | `100` (default), `Infinity` (to disable) | Integer value , causing the Kinesis API to start throwing exceptions to mimic the default shard limit. | | `KINESIS_ON_DEMAND_STREAM_COUNT_LIMIT` | `10` (default), `Infinity` (to disable) | Integer value , causing the Kinesis API to start throwing exceptions to mimic the default on demand stream count limit. | | `KINESIS_LATENCY` | `500` (default), `0` (to disable)| Integer value of milliseconds, causing the Kinesis API to delay returning a response in order to mimic latency from a live AWS call. | +| `KINESIS_MOCK_PROVIDER_ENGINE` | `node` (default) \| `scala` | String value of `node` (default) or `scala` that determines the underlying build of Kinesis Mock. | +| `KINESIS_MOCK_MAXIMUM_HEAP_SIZE` | `512m` (default) | JVM memory format string (default: `512m`) that sets the maximum memory size for the Kinesis Mock Scala server, corresponds to the JVM `-Xmx` flag. | +| `KINESIS_MOCK_INITIAL_HEAP_SIZE` | `256m` (default) | JVM memory format string that sets the initial memory size for the Kinesis Mock Scala server, corresponds to the JVM `-Xms` flag. | ### Lambda diff --git a/content/en/user-guide/aws/kinesis/index.md b/content/en/user-guide/aws/kinesis/index.md index 3eade63897..211137b86d 100644 --- a/content/en/user-guide/aws/kinesis/index.md +++ b/content/en/user-guide/aws/kinesis/index.md @@ -169,30 +169,20 @@ $ awslocal kinesis put-record \ You can fetch the CloudWatch logs for your Lambda function reading records from the stream, using AWS CLI or LocalStack Resource Browser. -## Configuration - -| Variable | Description | -| -------- | ----------- | -| `KINESIS_MOCK_PROVIDER_ENGINE` | String value of `node` (default) or `scala` that determines the underlying build of Kinesis Mock. [^scala-future] | -| `KINESIS_MOCK_MAXIMUM_HEAP_SIZE` | JVM memory format string (default: `512m`) that sets the maximum memory size for the Kinesis Mock Scala server, corresponds to the JVM `-Xmx` flag. [^scala] | -| `KINESIS_MOCK_INITIAL_HEAP_SIZE` | JVM memory format string (default: `256m`) that sets the initial memory size for the Kinesis Mock Scala server, corresponds to the JVM `-Xms` flag. [^scala] | -| `KINESIS_ERROR_PROBABILITY` | Decimal value between `0.0` (default) and `1.0`. This environment variable enables you to inject `ProvisionedThroughputException` at random intervals in your application. While this won't provide insight into your application's overall throughput handling, it aids in testing your application's exception-handling capabilities. | -| `KINESIS_SHARD_LIMIT` | Integer value (default: `100`) or `Infinity` (to disable). Use this variable to assess whether your application conforms to the assigned shard limit. Disabling this behavior requires explicitly setting `KINESIS_SHARD_LIMIT=Infinity`. | -| `KINESIS_LATENCY` | Integer value in milliseconds (default: `500`) or `0` (to disable). Particularly useful for testing latency-sensitive applications. Since local Kinesis service lacks latency simulation, you can introduce artificial latency into your AWS calls using this variable. To disable this behavior, set `KINESIS_LATENCY=0`. | -| `KINESIS_INITIALIZE_STREAMS` | **Deprecated.** Comma-delimited string with stream names, corresponding shard counts, and an optional region for initialization during startup. If no region is provided, the default region is used. For example, `KINESIS_INITIALIZE_STREAMS=my-first-stream:1,my-other-stream:2:us-west-2,my-last-stream:1`. In multi-account setups, the specified streams will be created for all accounts. | - ### Performance Tuning For high-volume workloads or large payloads, we recommend switching to the Scala engine via the `KINESIS_MOCK_PROVIDER_ENGINE=scala` flag, delivering up to 10x better performance compared to the default Node.js engine. Additionally, the following parameters can be tuned: -- Increase `KINESIS_MOCK_MAXIMUM_HEAP_SIZE` beyond the default `512m` to reduce JVM memory pressure. [^scala] -- Increase `KINESIS_MOCK_INITIAL_HEAP_SIZE` beyond the default `256m` to pre-allocate more JVM heap memory. [^scala] +- Increase `KINESIS_MOCK_MAXIMUM_HEAP_SIZE` beyond the default `512m` to reduce JVM memory pressure. +- Increase `KINESIS_MOCK_INITIAL_HEAP_SIZE` beyond the default `256m` to pre-allocate more JVM heap memory. - Reduce `KINESIS_LATENCY` artificial response delays from the default `500` milliseconds (or disable entirely with `0`). -[^scala]: This feature is only available in the more performant Scala Kinesis Mock build that can be toggled by setting `KINESIS_MOCK_PROVIDER_ENGINE=scala`. -[^scala-future]: Future versions of LocalStack will likely default to using the `scala` engine over the less-performant `node` version currently in use. +{{< note >}} +`KINESIS_MOCK_MAXIMUM_HEAP_SIZE` and `KINESIS_MOCK_INITIAL_HEAP_SIZE` are only applicable when using the Scala engine. +Future versions of LocalStack will likely default to using the `scala` engine over the less-performant `node` version currently in use. +{{< / note >}} ## Resource Browser From cbd637fc870003f2e7ec64a6e0bf34241074ca6a Mon Sep 17 00:00:00 2001 From: HarshCasper Date: Thu, 5 Jun 2025 06:56:15 -0700 Subject: [PATCH 3/5] fix --- content/en/references/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/references/configuration.md b/content/en/references/configuration.md index 86d8f6c908..e931703c95 100644 --- a/content/en/references/configuration.md +++ b/content/en/references/configuration.md @@ -221,7 +221,7 @@ Also see [OpenSearch configuration variables]({{< ref "configuration#opensearch" | `KINESIS_ON_DEMAND_STREAM_COUNT_LIMIT` | `10` (default), `Infinity` (to disable) | Integer value , causing the Kinesis API to start throwing exceptions to mimic the default on demand stream count limit. | | `KINESIS_LATENCY` | `500` (default), `0` (to disable)| Integer value of milliseconds, causing the Kinesis API to delay returning a response in order to mimic latency from a live AWS call. | | `KINESIS_MOCK_PROVIDER_ENGINE` | `node` (default) \| `scala` | String value of `node` (default) or `scala` that determines the underlying build of Kinesis Mock. | -| `KINESIS_MOCK_MAXIMUM_HEAP_SIZE` | `512m` (default) | JVM memory format string (default: `512m`) that sets the maximum memory size for the Kinesis Mock Scala server, corresponds to the JVM `-Xmx` flag. | +| `KINESIS_MOCK_MAXIMUM_HEAP_SIZE` | `512m` (default) | JVM memory format string that sets the maximum memory size for the Kinesis Mock Scala server, corresponds to the JVM `-Xmx` flag. | | `KINESIS_MOCK_INITIAL_HEAP_SIZE` | `256m` (default) | JVM memory format string that sets the initial memory size for the Kinesis Mock Scala server, corresponds to the JVM `-Xms` flag. | ### Lambda From edc826049c59c6fb4b5c14667f28adb7fe12ca33 Mon Sep 17 00:00:00 2001 From: HarshCasper Date: Thu, 5 Jun 2025 06:59:51 -0700 Subject: [PATCH 4/5] add config docs link --- content/en/user-guide/aws/kinesis/index.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/content/en/user-guide/aws/kinesis/index.md b/content/en/user-guide/aws/kinesis/index.md index 211137b86d..b6875da4b2 100644 --- a/content/en/user-guide/aws/kinesis/index.md +++ b/content/en/user-guide/aws/kinesis/index.md @@ -179,6 +179,8 @@ Additionally, the following parameters can be tuned: - Increase `KINESIS_MOCK_INITIAL_HEAP_SIZE` beyond the default `256m` to pre-allocate more JVM heap memory. - Reduce `KINESIS_LATENCY` artificial response delays from the default `500` milliseconds (or disable entirely with `0`). +Refer to our [Kinesis configuration documentation](https://docs.localstack.cloud/references/configuration/#kinesis) for more details on these parameters. + {{< note >}} `KINESIS_MOCK_MAXIMUM_HEAP_SIZE` and `KINESIS_MOCK_INITIAL_HEAP_SIZE` are only applicable when using the Scala engine. Future versions of LocalStack will likely default to using the `scala` engine over the less-performant `node` version currently in use. From be0052f5de75b4780897a1bece5f4beb955b5a13 Mon Sep 17 00:00:00 2001 From: HarshCasper Date: Thu, 5 Jun 2025 07:02:35 -0700 Subject: [PATCH 5/5] fix --- content/en/user-guide/aws/kinesis/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/user-guide/aws/kinesis/index.md b/content/en/user-guide/aws/kinesis/index.md index b6875da4b2..2cbac9f704 100644 --- a/content/en/user-guide/aws/kinesis/index.md +++ b/content/en/user-guide/aws/kinesis/index.md @@ -181,10 +181,10 @@ Additionally, the following parameters can be tuned: Refer to our [Kinesis configuration documentation](https://docs.localstack.cloud/references/configuration/#kinesis) for more details on these parameters. -{{< note >}} +{{< callout "note" >}} `KINESIS_MOCK_MAXIMUM_HEAP_SIZE` and `KINESIS_MOCK_INITIAL_HEAP_SIZE` are only applicable when using the Scala engine. Future versions of LocalStack will likely default to using the `scala` engine over the less-performant `node` version currently in use. -{{< / note >}} +{{< /callout >}} ## Resource Browser