From 5386ea99bb3eecf339a9b7167f4dee09d5fccd73 Mon Sep 17 00:00:00 2001 From: "R.I.Pienaar" Date: Mon, 23 Dec 2024 10:50:00 +0100 Subject: [PATCH 1/9] Refinements to MSG TTL spec during implementation Signed-off-by: R.I.Pienaar --- adr/ADR-43.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/adr/ADR-43.md b/adr/ADR-43.md index 042c343..960d5d1 100644 --- a/adr/ADR-43.md +++ b/adr/ADR-43.md @@ -22,7 +22,7 @@ Related issues [#3268](https://github.com/nats-io/nats-server/issues/3268) ## Per-Message TTL -We will allow a message to supply a TTL using a header called `Nats-TTL` followed by the duration as seconds. +We will allow a message to supply a TTL using a header called `Nats-TTL` followed by the duration as seconds or as a Go duration string like `1h`. The duration will be used by the server to calculate the deadline for removing the message based on its Stream timestamp and the stated duration. @@ -35,6 +35,8 @@ Setting the header `Nats-No-Expire` to `1` will result in a message that will ne A TTL of zero will be ignored, any other unparsable value will result in a error reported in the Pub Ack and the message being discarded. +When a message with the `Nats-TTL` header is published to a stream with the feature disabled the message will be rejected with an error. + ## Limit Tombstones Several scenarios for server-created tombstones can be imagined, the most often requested one though is when MaxAge @@ -70,3 +72,5 @@ type StreamConfig struct { LimitsTTL time.Duration `json:"limits_ttl"` } ``` + +When either these settings are set the Stream should require API level `1`. \ No newline at end of file From 1c87b0ff69028028fc8f25c78ba730353ba6b774 Mon Sep 17 00:00:00 2001 From: "R.I.Pienaar" Date: Thu, 9 Jan 2025 11:55:28 +0100 Subject: [PATCH 2/9] Change never expire behavior Signed-off-by: R.I.Pienaar --- adr/ADR-43.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/adr/ADR-43.md b/adr/ADR-43.md index 960d5d1..5e17dee 100644 --- a/adr/ADR-43.md +++ b/adr/ADR-43.md @@ -30,7 +30,7 @@ timestamp and the stated duration. The TTL may not exceed the Stream MaxAge. The shortest allowed TTL would be 1 second. When no specific TTL is given the MaxAge will apply. -Setting the header `Nats-No-Expire` to `1` will result in a message that will never be expired. +Setting the header `Nats-TTL` to `never` will result in a message that will never be expired. A TTL of zero will be ignored, any other unparsable value will result in a error reported in the Pub Ack and the message being discarded. From b9cd4a0ab0a2d77895c1b2fce218fb60eba50d49 Mon Sep 17 00:00:00 2001 From: "R.I.Pienaar" Date: Thu, 9 Jan 2025 12:25:16 +0100 Subject: [PATCH 3/9] Mention what can be edited Signed-off-by: R.I.Pienaar --- adr/ADR-43.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/adr/ADR-43.md b/adr/ADR-43.md index 5e17dee..f75b0f4 100644 --- a/adr/ADR-43.md +++ b/adr/ADR-43.md @@ -73,4 +73,6 @@ type StreamConfig struct { } ``` -When either these settings are set the Stream should require API level `1`. \ No newline at end of file +The `AllowMsgTTL` field must not be updatable, `LimitsTTL` may be updated. + +When either these settings are set the Stream should require API level `1`. From 3ef0d6a32436927b3b189d3c776d2968b3941ec5 Mon Sep 17 00:00:00 2001 From: "R.I.Pienaar" Date: Tue, 14 Jan 2025 14:16:52 +0100 Subject: [PATCH 4/9] Remove maxage boundary Signed-off-by: R.I.Pienaar --- adr/ADR-43.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/adr/ADR-43.md b/adr/ADR-43.md index f75b0f4..cedd8d8 100644 --- a/adr/ADR-43.md +++ b/adr/ADR-43.md @@ -27,9 +27,6 @@ We will allow a message to supply a TTL using a header called `Nats-TTL` followe The duration will be used by the server to calculate the deadline for removing the message based on its Stream timestamp and the stated duration. -The TTL may not exceed the Stream MaxAge. The shortest allowed TTL would be 1 second. When no specific TTL is given -the MaxAge will apply. - Setting the header `Nats-TTL` to `never` will result in a message that will never be expired. A TTL of zero will be ignored, any other unparsable value will result in a error reported in the Pub Ack and the message From 42019f10229c3a9a029b595bfda9461e27ce8383 Mon Sep 17 00:00:00 2001 From: "R.I.Pienaar" Date: Tue, 14 Jan 2025 15:49:51 +0100 Subject: [PATCH 5/9] mention LimitsTTL lower bound Signed-off-by: R.I.Pienaar --- adr/ADR-43.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/adr/ADR-43.md b/adr/ADR-43.md index cedd8d8..265f1b3 100644 --- a/adr/ADR-43.md +++ b/adr/ADR-43.md @@ -70,6 +70,6 @@ type StreamConfig struct { } ``` -The `AllowMsgTTL` field must not be updatable, `LimitsTTL` may be updated. +The `AllowMsgTTL` field must not be updatable, `LimitsTTL` may be updated and must have a minimum value of 1 second. When either these settings are set the Stream should require API level `1`. From 2719968655c577969e19107e67bcb4bf49927464 Mon Sep 17 00:00:00 2001 From: "R.I.Pienaar" Date: Thu, 16 Jan 2025 12:23:03 +0100 Subject: [PATCH 6/9] Clarify sources/mirrors Signed-off-by: R.I.Pienaar --- .gitignore | 1 + adr/ADR-43.md | 13 ++++++++++++- 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index 485dee6..090a1f0 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,2 @@ .idea +.DS_Store diff --git a/adr/ADR-43.md b/adr/ADR-43.md index 265f1b3..a5d3882 100644 --- a/adr/ADR-43.md +++ b/adr/ADR-43.md @@ -4,7 +4,7 @@ |----------|---------------------------------| | Date | 2024-07-11 | | Author | @ripienaar | -| Status | Approved | +| Status | Implemented | | Tags | jetstream, client, server, 2.11 | ## Context and motivation @@ -22,6 +22,8 @@ Related issues [#3268](https://github.com/nats-io/nats-server/issues/3268) ## Per-Message TTL +### General Behavior + We will allow a message to supply a TTL using a header called `Nats-TTL` followed by the duration as seconds or as a Go duration string like `1h`. The duration will be used by the server to calculate the deadline for removing the message based on its Stream @@ -34,6 +36,14 @@ being discarded. When a message with the `Nats-TTL` header is published to a stream with the feature disabled the message will be rejected with an error. +### Sources and Mirrors + +When messages arrive over a Source or a Mirror processing is a little bit different: Where for a normal client-published message we would reject it if the stream lacks the feature from a Source/Mirror we would accept it. + +This allow one to make archives of record that would include these messages, however if the Source or Mirror has the `AllowMsgTTL` set those message with the header will be processed as above. + +Sources may set the `LimitsTTL` option but Mirrors may not since the `LimitsTTL` behavior will insert new messages into the Stream it might make it impossible to match sequences in the 2 Mirrors. + ## Limit Tombstones Several scenarios for server-created tombstones can be imagined, the most often requested one though is when MaxAge @@ -71,5 +81,6 @@ type StreamConfig struct { ``` The `AllowMsgTTL` field must not be updatable, `LimitsTTL` may be updated and must have a minimum value of 1 second. +The `LimitsTTL` setting may not be set on a Mirror Stream. When either these settings are set the Stream should require API level `1`. From 91317ac14380ddee664690af8f922d18686405c4 Mon Sep 17 00:00:00 2001 From: "R.I.Pienaar" Date: Fri, 17 Jan 2025 10:10:28 +0100 Subject: [PATCH 7/9] rewrite source/mirror section Signed-off-by: R.I.Pienaar --- adr/ADR-43.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/adr/ADR-43.md b/adr/ADR-43.md index a5d3882..f28357e 100644 --- a/adr/ADR-43.md +++ b/adr/ADR-43.md @@ -36,14 +36,6 @@ being discarded. When a message with the `Nats-TTL` header is published to a stream with the feature disabled the message will be rejected with an error. -### Sources and Mirrors - -When messages arrive over a Source or a Mirror processing is a little bit different: Where for a normal client-published message we would reject it if the stream lacks the feature from a Source/Mirror we would accept it. - -This allow one to make archives of record that would include these messages, however if the Source or Mirror has the `AllowMsgTTL` set those message with the header will be processed as above. - -Sources may set the `LimitsTTL` option but Mirrors may not since the `LimitsTTL` behavior will insert new messages into the Stream it might make it impossible to match sequences in the 2 Mirrors. - ## Limit Tombstones Several scenarios for server-created tombstones can be imagined, the most often requested one though is when MaxAge @@ -61,6 +53,14 @@ The `Nats-Limit-Applied` field is there to support future expansion of this feat This behaviour is off by default unless opted in on the Stream Configuration. +### Sources and Mirrors + +Sources and Mirrors will always accept and store messages with `Nats-TTL` header present, even if the `AllowMsgTTL` setting is disabled in the Stream settings. + +If the `AllowMsgTTL` setting is enabled then processing continues as outlined in the General Behavior section with messages removed after the TTL. With the setting disabled the messages are just stored. + +Sources may set the `LimitsTTL` option and processing of messages with the `Nats-TTL` will place tombstones, but, Mirrors may not enable `LimitsTTL` since it would insert new messages into the Stream it might make it impossible to match sequences from the Mirrored Stream. + ## Stream Configuration Weather or not a stream support this behavior should be a configuration opt-in. We want clients to definitely know From cdcb864da9fdbe042b5333fbbf9a5f4441ecae34 Mon Sep 17 00:00:00 2001 From: "R.I.Pienaar" Date: Tue, 21 Jan 2025 11:06:48 +0100 Subject: [PATCH 8/9] split delete markers into two settings Signed-off-by: R.I.Pienaar --- adr/ADR-43.md | 39 ++++++++++++++++++++------------------- 1 file changed, 20 insertions(+), 19 deletions(-) diff --git a/adr/ADR-43.md b/adr/ADR-43.md index f28357e..2df5b93 100644 --- a/adr/ADR-43.md +++ b/adr/ADR-43.md @@ -9,8 +9,7 @@ ## Context and motivation -Streams support a one-size-fits-all approach to message TTL based on the MaxAge setting. This causes any message in the -Stream to expire at that age. +Streams support a one-size-fits-all approach to message TTL based on the MaxAge setting. This causes any message in the Stream to expire at that age. There are numerous uses for a per-message version of this limit, some listed below: @@ -26,8 +25,7 @@ Related issues [#3268](https://github.com/nats-io/nats-server/issues/3268) We will allow a message to supply a TTL using a header called `Nats-TTL` followed by the duration as seconds or as a Go duration string like `1h`. -The duration will be used by the server to calculate the deadline for removing the message based on its Stream -timestamp and the stated duration. +The duration will be used by the server to calculate the deadline for removing the message based on its Stream timestamp and the stated duration. Setting the header `Nats-TTL` to `never` will result in a message that will never be expired. @@ -38,11 +36,9 @@ When a message with the `Nats-TTL` header is published to a stream with the feat ## Limit Tombstones -Several scenarios for server-created tombstones can be imagined, the most often requested one though is when MaxAge -removes last value (ie. the current value) for a Key. +Several scenarios for server-created tombstones can be imagined, the most often requested one though is when MaxAge removes last value (ie. the current value) for a Key. -In this case when the server removes a message and the message is the last in the subject it would place a message -with a TTL matching the Stream configuration value. The following headers would be placed: +In this case when the server removes a message and the message is the last in the subject it would place a message with a TTL matching the Stream configuration value. The following headers would be placed: ``` Nats-Applied-Limit: MaxAge @@ -59,28 +55,33 @@ Sources and Mirrors will always accept and store messages with `Nats-TTL` header If the `AllowMsgTTL` setting is enabled then processing continues as outlined in the General Behavior section with messages removed after the TTL. With the setting disabled the messages are just stored. -Sources may set the `LimitsTTL` option and processing of messages with the `Nats-TTL` will place tombstones, but, Mirrors may not enable `LimitsTTL` since it would insert new messages into the Stream it might make it impossible to match sequences from the Mirrored Stream. +Sources may set the `SubjectDeleteMarkers` option and processing of messages with the `Nats-TTL` will place tombstones, but, Mirrors may not enable `SubjectDeleteMarkers` since it would insert new messages into the Stream it might make it impossible to match sequences from the Mirrored Stream. ## Stream Configuration -Weather or not a stream support this behavior should be a configuration opt-in. We want clients to definitely know -when this is supported which the opt-in approach with a boolean on the configuration would make clear. +Weather or not a stream support this behavior should be a configuration opt-in. We want clients to definitely know when this is supported which the opt-in approach with a boolean on the configuration would make clear. -We have to assume someone will want to create a replication topology where at some point in the topology these tombstone -type messages are retained for an audit trail. So a Stream with this feature enabled can replicate to one with it -disabled and all the messages that would have been TTLed will be retained. +We have to assume someone will want to create a replication topology where at some point in the topology these tombstone type messages are retained for an audit trail. So a Stream with this feature enabled can replicate to one with it disabled and all the messages that would have been TTLed will be retained. ```golang type StreamConfig struct { // AllowMsgTTL allows header initiated per-message TTLs AllowMsgTTL bool `json:"allow_msg_ttl"` - // LimitsTTL activates writing of messages when limits are applied with a specific TTL - LimitsTTL time.Duration `json:"limits_ttl"` + // Enables placing markers in the stream for certain message delete operations + SubjectDeleteMarkers bool `json:"subject_delete_markers,omitempty"` + // When placing a marker, how long should it be valid, defaults to 15m + SubjectDeleteMarkerTTL string `json:"subject_delete_marker_ttl,omitempty"` } ``` -The `AllowMsgTTL` field must not be updatable, `LimitsTTL` may be updated and must have a minimum value of 1 second. -The `LimitsTTL` setting may not be set on a Mirror Stream. +Restrictions: + + * The `AllowMsgTTL` field must not be updatable. + * The `AllowMsgTTL` and `SubjectDeleteMarkerTTL` has a minimum value of 1 second. + * The `SubjectDeleteMarkers` setting may not be set on a Mirror Stream. + * The `SubjectDeleteMarkers` setting requires `AllowMsgTTL` and must error when not set. + * The `SubjectDeleteMarkerTTL` may only be set when `SubjectDeleteMarkers` is set. + * When `SubjectDeleteMarkerTTL` is unset the server will use `15m` as the default and will not update the supplied configuration, future info requests will show the setting blank. This will have the effect of updating already deployed Streams should we change the default. + * When `AllowMsgTTL` or `SubjectDeleteMarkers` are set the Stream should require API level `1`. -When either these settings are set the Stream should require API level `1`. From 68de1a82322822defa82d0a9bb82c72a76196fd1 Mon Sep 17 00:00:00 2001 From: "R.I.Pienaar" Date: Wed, 22 Jan 2025 09:16:36 +0100 Subject: [PATCH 9/9] Update for pedantic mode Signed-off-by: R.I.Pienaar --- adr/ADR-43.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/adr/ADR-43.md b/adr/ADR-43.md index 2df5b93..3b655a2 100644 --- a/adr/ADR-43.md +++ b/adr/ADR-43.md @@ -82,6 +82,7 @@ Restrictions: * The `SubjectDeleteMarkers` setting may not be set on a Mirror Stream. * The `SubjectDeleteMarkers` setting requires `AllowMsgTTL` and must error when not set. * The `SubjectDeleteMarkerTTL` may only be set when `SubjectDeleteMarkers` is set. - * When `SubjectDeleteMarkerTTL` is unset the server will use `15m` as the default and will not update the supplied configuration, future info requests will show the setting blank. This will have the effect of updating already deployed Streams should we change the default. + * When `SubjectDeleteMarkerTTL` is unset the server will use `15m` as the default and will update the supplied configuration. + * When `SubjectDeleteMarkerTTL` is not given with `SubjectDeleteMarkers` set in Pedantic mode no default will be set and the request will fail. * When `AllowMsgTTL` or `SubjectDeleteMarkers` are set the Stream should require API level `1`.