Release/3.1.0

CHANGELOG

@@ -1,3 +1,16 @@
+Version 3.1.0 (2025-01-20)
+--------------------------
+Change License to SLULA 1.1
+Bump alpine version in Dockerfile
+More improvements in HTTP client
+Bump all dependencies and use latest localstack (#389)
+Add option to set kinsumer client name
+Improvements to HTTP Connection Pooling (#385)
+Update kinsumer to v1.4.0
+Add configuration settings for PubSub source (#380)
+Add E2E latency metrics
+Configurable time layout in `epoch`/`epochMillis` JQ functions
+
 Version 3.0.0 (2024-11-06)
 --------------------------
 Ensure all timestamps used in reporting are UTC

Dockerfile.aws

@@ -1,4 +1,4 @@
-FROM --platform=$TARGETPLATFORM alpine:3.20.0
+FROM --platform=$TARGETPLATFORM alpine:3.21.2
 LABEL org.opencontainers.image.authors="Snowplow Analytics <support@snowplow.io>"
 
 ARG TARGETARCH

Dockerfile.main

@@ -1,4 +1,4 @@
-FROM --platform=$TARGETPLATFORM alpine:3.20.0
+FROM --platform=$TARGETPLATFORM alpine:3.21.2
 LABEL org.opencontainers.image.authors="Snowplow Analytics <support@snowplow.io>"
 
 ARG TARGETARCH

LICENSE.md

@@ -1,18 +1,20 @@
 # Snowplow Limited Use License Agreement
 
-_Version 1.0, January 2024_
+_Version 1.1, November, 2024_
 
-This Snowplow Limited Use License Agreement, Version 1.0 (the “Agreement”) sets forth the terms on which Snowplow Analytics, Ltd. (“Snowplow”) makes available certain software (the “Software”). BY INSTALLING, DOWNLOADING, ACCESSING, OR USING ANY OF THE SOFTWARE, YOU AGREE TO THE TERMS AND CONDITIONS OF THIS AGREEMENT. IF YOU DO NOT AGREE TO SUCH TERMS AND CONDITIONS, YOU MUST NOT USE THE SOFTWARE. IF YOU ARE RECEIVING THE SOFTWARE ON BEHALF OF A LEGAL ENTITY, YOU REPRESENT AND WARRANT THAT YOU HAVE THE ACTUAL AUTHORITY TO AGREE TO THE TERMS AND CONDITIONS OF THIS AGREEMENT ON BEHALF OF SUCH ENTITY. “Licensee” means you, an individual, or the entity on whose behalf you are receiving the Software.
+This Snowplow Limited Use License Agreement, Version 1.1 (the “Agreement”) sets forth the terms on which Snowplow Analytics, Ltd. (“Snowplow”) makes available certain software (the “Software”). BY INSTALLING, DOWNLOADING, ACCESSING, OR USING ANY OF THE SOFTWARE, YOU AGREE TO THE TERMS AND CONDITIONS OF THIS AGREEMENT. IF YOU DO NOT AGREE TO SUCH TERMS AND CONDITIONS, YOU MUST NOT USE THE SOFTWARE. IF YOU ARE RECEIVING THE SOFTWARE ON BEHALF OF A LEGAL ENTITY, YOU REPRESENT AND WARRANT THAT YOU HAVE THE ACTUAL AUTHORITY TO AGREE TO THE TERMS AND CONDITIONS OF THIS AGREEMENT ON BEHALF OF SUCH ENTITY. “Licensee” means you, an individual, or the entity on whose behalf you are receiving the Software.
 
-## LICENSE GRANT AND CONDITIONS
+## 1. LICENSE GRANT AND CONDITIONS
 
-**1.1 License.** Subject to the terms and conditions of this Agreement, Snowplow hereby grants to Licensee a non-exclusive, royalty-free, worldwide, non-transferable, non-sublicensable license during the term of this Agreement to: (a) use the Software; (b) prepare modifications and derivative works of the Software; and (c) reproduce copies of the Software (the “License”). No right to distribute or make available the Software is granted under this License. Licensee is not granted the right to, and Licensee shall not, exercise the License for any Excluded Purpose.
+**1.1 License.** Subject to the terms and conditions of this Agreement, Snowplow hereby grants to Licensee a non-exclusive, royalty-free, worldwide, non-transferable, non-sublicensable license during the term of this Agreement to: (a) use the Software; (b) prepare modifications and derivative works of the Software; and (c) reproduce copies of the Software (the “License”). No right to distribute or make available the Software is granted under this License. Licensee is not granted the right to, and Licensee shall not, exercise the License for any Competing Use, and Licensee may exercise the License only for Non-Production Use or Non-Commercial Use.
 
-**1.2** For purposes of this Agreement, an “Excluded Purpose” is any use that is either a Competing Use or a Highly-Available Production Use, or both of them.
+**1.2 Definitions.** For purposes of this Agreement:
 
-* **1.2.1** A “Competing Use” is making available any on-premises or distributed software product, or any software-as-a-service, platform-as-a-service, infrastructure-as-a-service, or other similar online service, that competes with any products or services that Snowplow or any of its affiliates provides using the Software.
+* **1.2.1** “Competing Use” is making available any on-premises or distributed software product, or any software-as-a-service, platform-as-a-service, infrastructure-as-a-service, or other similar online service, that competes with any products or services that Snowplow or any of its affiliates provides using the Software.
 
-* **1.2.2** Highly-Available Production Use is any highly-available use, including without limitation any use where multiple instances of any Software component run concurrently to avoid a single point of failure, in a production environment, where production means use on live data.
+* **1.2.2** “Non-Production Use” means any use of the Software to process test or synthetic data to evaluate the sufficiency of the Software for use by Licensee.
+
+* **1.2.3** “Non-Commercial Use” is only: (a) personal use for research, experiment, personal study, or hobby projects, without any anticipated commercial application, or (b) use for teaching purposes by lecturers of a school or university.
 
 **1.3 Conditions.** In consideration of the License, Licensee’s use of the Software is subject to the following conditions:
 
@@ -22,8 +24,8 @@ This Snowplow Limited Use License Agreement, Version 1.0 (the “Agreement”) s
 
   ```
   This software is made available by Snowplow Analytics, Ltd.,
-  under the terms of the Snowplow Limited Use License Agreement, Version 1.0
-  located at https://docs.snowplow.io/limited-use-license-1.0
+  under the terms of the Snowplow Limited Use License Agreement, Version 1.1
+  located at https://docs.snowplow.io/limited-use-license-1.1
   BY INSTALLING, DOWNLOADING, ACCESSING, USING OR DISTRIBUTING ANY PORTION
   OF THE SOFTWARE, YOU AGREE TO THE TERMS OF SUCH LICENSE AGREEMENT.
   ```
@@ -32,23 +34,23 @@ This Snowplow Limited Use License Agreement, Version 1.0 (the “Agreement”) s
 
 **1.5 No Sublicensing.** The License does not include the right to sublicense the Software, however, each recipient to which Licensee provides the Software may exercise the Licenses so long as such recipient agrees to the terms and conditions of this Agreement.
 
-## TERM AND TERMINATION
+## 2. TERM AND TERMINATION
 
 This Agreement will continue unless and until earlier terminated as set forth herein. If Licensee breaches any of its conditions or obligations under this Agreement, this Agreement will terminate automatically and the License will terminate automatically and permanently.
 
-## INTELLECTUAL PROPERTY
+## 3. INTELLECTUAL PROPERTY
 
 As between the parties, Snowplow will retain all right, title, and interest in the Software, and all intellectual property rights therein. Snowplow hereby reserves all rights not expressly granted to Licensee in this Agreement. Snowplow hereby reserves all rights in its trademarks and service marks, and no licenses therein are granted in this Agreement.
 
-## DISCLAIMER
+## 4. DISCLAIMER
 
 SNOWPLOW HEREBY DISCLAIMS ANY AND ALL WARRANTIES AND CONDITIONS, EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, AND SPECIFICALLY DISCLAIMS ANY WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, WITH RESPECT TO THE SOFTWARE.
 
-## LIMITATION OF LIABILITY
+## 5. LIMITATION OF LIABILITY
 
 SNOWPLOW WILL NOT BE LIABLE FOR ANY DAMAGES OF ANY KIND, INCLUDING BUT NOT LIMITED TO LOST PROFITS OR ANY CONSEQUENTIAL, SPECIAL, INCIDENTAL, INDIRECT, OR DIRECT DAMAGES, HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, ARISING OUT OF THIS AGREEMENT. THE FOREGOING SHALL APPLY TO THE EXTENT PERMITTED BY APPLICABLE LAW.
 
-## GENERAL
+## 6. GENERAL
 
 **6.1 Governing Law.** This Agreement will be governed by and interpreted in accordance with the laws of the state of Delaware, without reference to its conflict of laws principles. If Licensee is located within the United States, all disputes arising out of this Agreement are subject to the exclusive jurisdiction of courts located in Delaware, USA. If Licensee is located outside of the United States, any dispute, controversy or claim arising out of or relating to this Agreement will be referred to and finally determined by arbitration in accordance with the JAMS International Arbitration Rules. The tribunal will consist of one arbitrator. The place of arbitration will be in the State of Delaware, USA. The language to be used in the arbitral proceedings will be English. Judgment upon the award rendered by the arbitrator may be entered in any court having jurisdiction thereof.
 

README.md

@@ -12,11 +12,11 @@ See the [documention](https://docs.snowplow.io/docs/destinations/forwarding-even
 
 Copyright (c) 2020-present Snowplow Analytics Ltd. All rights reserved.
 
-The default distribution contains everything except for the Kinesis source, i.e. the ability to read from AWS Kinesis. This distribution is all licensed under the [Snowplow Community License](https://docs.snowplow.io/community-license-1.0). _(If you are uncertain how it applies to your use case, check our answers to [frequently asked questions](https://docs.snowplow.io/docs/contributing/community-license-faq/).)_
+The default distribution contains everything except for the Kinesis source, i.e. the ability to read from AWS Kinesis. This distribution is all licensed under the [Snowplow Limited Use License](https://docs.snowplow.io/limited-use-license-1.1/). _(If you are uncertain how it applies to your use case, check our answers to [frequently asked questions](https://docs.snowplow.io/docs/resources/limited-use-license-faq/).)_
 
-The AWS-specific distribution contains everything, including the Kinesis source, i.e. the ability to read from AWS Kinesis. Like the default distribution, it’s licensed under the [Snowplow Community License](https://docs.snowplow.io/community-license-1.0) ([frequently asked questions](https://docs.snowplow.io/docs/contributing/community-license-faq/)). However, this distribution has a dependency on [twitchscience/kinsumer](https://github.com/twitchscience/kinsumer), which is licensed by Twitch under the [Amazon Software License](https://github.com/twitchscience/kinsumer/blob/master/LICENSE).
+The AWS-specific distribution contains everything, including the Kinesis source, i.e. the ability to read from AWS Kinesis. Like the default distribution, it’s licensed under the [Snowplow Limited Use License](https://docs.snowplow.io/limited-use-license-1.1/) ([frequently asked questions](https://docs.snowplow.io/docs/resources/limited-use-license-faq/)). However, this distribution has a dependency on [twitchscience/kinsumer](https://github.com/twitchscience/kinsumer), which is licensed by Twitch under the [Amazon Software License](https://github.com/twitchscience/kinsumer/blob/master/LICENSE).
 
 To comply with the [Amazon Software License](https://github.com/twitchscience/kinsumer/blob/master/LICENSE), you may only use this distribution of Snowbridge _“with the web services, computing platforms or applications provided by Amazon.com, Inc. or its affiliates, including Amazon Web Services, Inc.”_
 
-[release-image]: http://img.shields.io/badge/golang-3.0.0-6ad7e5.svg?style=flat
+[release-image]: http://img.shields.io/badge/golang-3.1.0-6ad7e5.svg?style=flat
 [releases]: https://github.com/snowplow/snowbridge/releases/

VERSION

@@ -1 +1 @@
-3.0.0
+3.1.0

assets/docs/configuration/metrics/e2e-latency-example.hcl

@@ -0,0 +1,4 @@
+metrics {
+  # Optional toggle for E2E latency (difference between Snowplow collector timestamp and target write timestamp)
+  enable_e2e_latency = true
+}

assets/docs/configuration/overview-full-example.hcl

@@ -108,6 +108,11 @@ retry {
   }
 }
 
+metrics {
+  # Optional toggle for E2E latency (difference between Snowplow collector timestamp and target write timestamp)
+  enable_e2e_latency = true
+}
+
 license {
   accept = true
 }

assets/docs/configuration/sources/kinesis-full-example.hcl

@@ -36,5 +36,8 @@ source {
 
     # Maximum concurrent goroutines (lightweight threads) for message processing (default: 50)
     concurrent_writes = 15
+
+    # The name of the Kinesis client that is used to allocate shards. It must be unique per instance of Snowbridge.
+    client_name = env.HOSTNAME
   }
 }

assets/docs/configuration/sources/pubsub-full-example.hcl

@@ -8,7 +8,24 @@ source {
     # subscription ID for the pubsub subscription
     subscription_id   = "subscription-id"
 
-    # Maximum concurrent goroutines (lightweight threads) for message processing (default: 50)
-    concurrent_writes = 20
+    # This option is deprecated for the pubsub source, and will be changed or removed in the next major release.
+    # Use streaming_pull_goroutines, max_outstanding_messages, and max_outstanding_bytes to configure concurrency instead.
+    # Where streaming_pull_goroutines is set, this option is ignored.
+    concurrent_writes = 1
+
+    # Maximum number of unprocessed messages (default 1000)
+    max_outstanding_messages = 2000
+
+    # Maximum size of unprocessed messages (default 1e9)
+    max_outstanding_bytes = 2e9
+
+    # Minimum ack extension period when a message is received
+    min_extension_period_seconds = 10
+
+    # Number of streaming pull connections to open at once
+    streaming_pull_goroutines = 1
+
+    # Configures the GRPC connection pool size of the pubsub client
+    grpc_connection_pool_size = 4
   }
-}
+}

assets/test/source/configs/source-kinesis-extended.hcl

@@ -8,5 +8,6 @@ source {
     app_name          = "testApp"
     start_timestamp   = "2022-03-15 07:52:53"
     concurrent_writes = 51
+    client_name       = "test_client_name"
   }
 }

assets/test/transformconfig/TestEnginesAndTransformations/configs/transform-collector-tstamp.hcl

@@ -0,0 +1,3 @@
+metrics {
+  enable_e2e_latency = true
+}

assets/vars.go

@@ -3,8 +3,8 @@
  * All rights reserved.
  *
  * This software is made available by Snowplow Analytics, Ltd.,
- * under the terms of the Snowplow Limited Use License Agreement, Version 1.0
- * located at https://docs.snowplow.io/limited-use-license-1.0
+ * under the terms of the Snowplow Limited Use License Agreement, Version 1.1
+ * located at https://docs.snowplow.io/limited-use-license-1.1
  * BY INSTALLING, DOWNLOADING, ACCESSING, USING OR DISTRIBUTING ANY PORTION
  * OF THE SOFTWARE, YOU AGREE TO THE TERMS OF SUCH LICENSE AGREEMENT.
  */

cmd/aws/cli/main.go

@@ -3,8 +3,8 @@
  * All rights reserved.
  *
  * This software is made available by Snowplow Analytics, Ltd.,
- * under the terms of the Snowplow Limited Use License Agreement, Version 1.0
- * located at https://docs.snowplow.io/limited-use-license-1.0
+ * under the terms of the Snowplow Limited Use License Agreement, Version 1.1
+ * located at https://docs.snowplow.io/limited-use-license-1.1
  * BY INSTALLING, DOWNLOADING, ACCESSING, USING OR DISTRIBUTING ANY PORTION
  * OF THE SOFTWARE, YOU AGREE TO THE TERMS OF SUCH LICENSE AGREEMENT.
  */

cmd/cli/cli.go

@@ -3,8 +3,8 @@
  * All rights reserved.
  *
  * This software is made available by Snowplow Analytics, Ltd.,
- * under the terms of the Snowplow Limited Use License Agreement, Version 1.0
- * located at https://docs.snowplow.io/limited-use-license-1.0
+ * under the terms of the Snowplow Limited Use License Agreement, Version 1.1
+ * located at https://docs.snowplow.io/limited-use-license-1.1
  * BY INSTALLING, DOWNLOADING, ACCESSING, USING OR DISTRIBUTING ANY PORTION
  * OF THE SOFTWARE, YOU AGREE TO THE TERMS OF SUCH LICENSE AGREEMENT.
  */

cmd/cli/cli_test.go

@@ -3,8 +3,8 @@
  * All rights reserved.
  *
  * This software is made available by Snowplow Analytics, Ltd.,
- * under the terms of the Snowplow Limited Use License Agreement, Version 1.0
- * located at https://docs.snowplow.io/limited-use-license-1.0
+ * under the terms of the Snowplow Limited Use License Agreement, Version 1.1
+ * located at https://docs.snowplow.io/limited-use-license-1.1
  */
 
 package cli

cmd/constants.go

@@ -3,8 +3,8 @@
  * All rights reserved.
  *
  * This software is made available by Snowplow Analytics, Ltd.,
- * under the terms of the Snowplow Limited Use License Agreement, Version 1.0
- * located at https://docs.snowplow.io/limited-use-license-1.0
+ * under the terms of the Snowplow Limited Use License Agreement, Version 1.1
+ * located at https://docs.snowplow.io/limited-use-license-1.1
  * BY INSTALLING, DOWNLOADING, ACCESSING, USING OR DISTRIBUTING ANY PORTION
  * OF THE SOFTWARE, YOU AGREE TO THE TERMS OF SUCH LICENSE AGREEMENT.
  */
@@ -13,7 +13,7 @@ package cmd
 
 const (
 	// AppVersion is the current version of the app
-	AppVersion = "3.0.0"
+	AppVersion = "3.1.0"
 
 	// AppName is the name of the application to use in logging / places that require the artifact
 	AppName = "snowbridge"

cmd/init.go

@@ -3,8 +3,8 @@
  * All rights reserved.
  *
  * This software is made available by Snowplow Analytics, Ltd.,
- * under the terms of the Snowplow Limited Use License Agreement, Version 1.0
- * located at https://docs.snowplow.io/limited-use-license-1.0
+ * under the terms of the Snowplow Limited Use License Agreement, Version 1.1
+ * located at https://docs.snowplow.io/limited-use-license-1.1
  * BY INSTALLING, DOWNLOADING, ACCESSING, USING OR DISTRIBUTING ANY PORTION
  * OF THE SOFTWARE, YOU AGREE TO THE TERMS OF SUCH LICENSE AGREEMENT.
  */

cmd/init_test.go

@@ -3,8 +3,8 @@
  * All rights reserved.
  *
  * This software is made available by Snowplow Analytics, Ltd.,
- * under the terms of the Snowplow Limited Use License Agreement, Version 1.0
- * located at https://docs.snowplow.io/limited-use-license-1.0
+ * under the terms of the Snowplow Limited Use License Agreement, Version 1.1
+ * located at https://docs.snowplow.io/limited-use-license-1.1
  * BY INSTALLING, DOWNLOADING, ACCESSING, USING OR DISTRIBUTING ANY PORTION
  * OF THE SOFTWARE, YOU AGREE TO THE TERMS OF SUCH LICENSE AGREEMENT.
  */

cmd/main/cli/main.go

@@ -3,8 +3,8 @@
  * All rights reserved.
  *
  * This software is made available by Snowplow Analytics, Ltd.,
- * under the terms of the Snowplow Limited Use License Agreement, Version 1.0
- * located at https://docs.snowplow.io/limited-use-license-1.0
+ * under the terms of the Snowplow Limited Use License Agreement, Version 1.1
+ * located at https://docs.snowplow.io/limited-use-license-1.1
  * BY INSTALLING, DOWNLOADING, ACCESSING, USING OR DISTRIBUTING ANY PORTION
  * OF THE SOFTWARE, YOU AGREE TO THE TERMS OF SUCH LICENSE AGREEMENT.
  */

config/component.go

@@ -3,8 +3,8 @@
  * All rights reserved.
  *
  * This software is made available by Snowplow Analytics, Ltd.,
- * under the terms of the Snowplow Limited Use License Agreement, Version 1.0
- * located at https://docs.snowplow.io/limited-use-license-1.0
+ * under the terms of the Snowplow Limited Use License Agreement, Version 1.1
+ * located at https://docs.snowplow.io/limited-use-license-1.1
  * BY INSTALLING, DOWNLOADING, ACCESSING, USING OR DISTRIBUTING ANY PORTION
  * OF THE SOFTWARE, YOU AGREE TO THE TERMS OF SUCH LICENSE AGREEMENT.
  */

config/component_test.go

@@ -3,8 +3,8 @@
  * All rights reserved.
  *
  * This software is made available by Snowplow Analytics, Ltd.,
- * under the terms of the Snowplow Limited Use License Agreement, Version 1.0
- * located at https://docs.snowplow.io/limited-use-license-1.0
+ * under the terms of the Snowplow Limited Use License Agreement, Version 1.1
+ * located at https://docs.snowplow.io/limited-use-license-1.1
  * BY INSTALLING, DOWNLOADING, ACCESSING, USING OR DISTRIBUTING ANY PORTION
  * OF THE SOFTWARE, YOU AGREE TO THE TERMS OF SUCH LICENSE AGREEMENT.
  */