Release Notes for version 2.8 #4538
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>
kkondaka
requested review from
chenqi0805,
engechas,
graytaylor0,
dinujoh,
asifsmohammed,
KarstenSchnitter,
dlvenable and
oeyh
as code owners
dlvenable
requested changes
May 16, 2024
| --- | ||
|
|
||
| ### Features | ||
| * Support conditional expression to evaluate based on the data type for a given field ([#4478](https://github.com/opensearch-project/data-prepper/issues/4478)) |
There was a problem hiding this comment.
Please add this issue: #4534
And let's start with it. It is our biggest feature in the release.
| * Add system property to allow pausing and resuming s3 scan worker thread processing, set this property to pause on documentdb leader ([#4495](https://github.com/opensearch-project/data-prepper/pull/4495)) | ||
| * Add new S3 source option and modify docdb template ([#4492](https://github.com/opensearch-project/data-prepper/pull/4492)) | ||
| * Move MongoDB/DocDB stream checkpoint to separate thread ([#4477](https://github.com/opensearch-project/data-prepper/pull/4477)) | ||
| * HTTP data chunking support for kafka buffer ([#4475](https://github.com/opensearch-project/data-prepper/pull/4475)) |
There was a problem hiding this comment.
We highlight this in the blog, so let's move it to the top of the "Enhancements" section.
| * Initial work to support core data types in Data Prepper ([#4496](https://github.com/opensearch-project/data-prepper/pull/4496)) | ||
| * Add system property to allow pausing and resuming s3 scan worker thread processing, set this property to pause on documentdb leader ([#4495](https://github.com/opensearch-project/data-prepper/pull/4495)) | ||
| * Add new S3 source option and modify docdb template ([#4492](https://github.com/opensearch-project/data-prepper/pull/4492)) | ||
| * Move MongoDB/DocDB stream checkpoint to separate thread ([#4477](https://github.com/opensearch-project/data-prepper/pull/4477)) |
There was a problem hiding this comment.
I think all of these "docdb"/"mongodb"/"documentdb" changes can be removed from here. This feature is new, so we can just rely on the one issue to highlight the feature in the top (see my other comment about that).
| * Add creation and aggregation of dynamic S3 groups based on events ([#4346](https://github.com/opensearch-project/data-prepper/pull/4346)) | ||
| * Refactor PipelinesDataFlowModelParser to take in an InputStream instead of a file path ([#4289](https://github.com/opensearch-project/data-prepper/pull/4289)) | ||
| * Add support to use old ddb stream image for REMOVE events ([#4275](https://github.com/opensearch-project/data-prepper/pull/4275)) | ||
| * ENH: automatic credential refresh in kafka source ([#4258](https://github.com/opensearch-project/data-prepper/pull/4258)) |
There was a problem hiding this comment.
Let's move this up to the top of this section. We discuss it in the blog.
| --- | ||
|
|
||
| ### Features | ||
| * Support conditional expression to evaluate based on the data type for a given field ([#4478](https://github.com/opensearch-project/data-prepper/issues/4478)) |
There was a problem hiding this comment.
Also, I don't see #3310. This is another big feature and should be toward the top of the features list.
| * Support encoding JSON ([#832](https://github.com/opensearch-project/data-prepper/issues/832)) | ||
| * Write json processor ([#4514](https://github.com/opensearch-project/data-prepper/pull/4514)) | ||
| * Support Mongo/DocDB stream Delete and other Operation types ([#4497](https://github.com/opensearch-project/data-prepper/pull/4497)) | ||
| * Support default mapping for Mongo/DocumentDB data types ([#4499](https://github.com/opensearch-project/data-prepper/pull/4499)) |
There was a problem hiding this comment.
No need for this one:
Suggested change
| * Support default mapping for Mongo/DocumentDB data types ([#4499](https://github.com/opensearch-project/data-prepper/pull/4499)) |
| --- | ||
|
|
||
| ### Features | ||
| * Support conditional expression to evaluate based on the data type for a given field ([#4478](https://github.com/opensearch-project/data-prepper/issues/4478)) |
There was a problem hiding this comment.
Please also include #2700 as a feature.
| --- | ||
|
|
||
| ### Features | ||
| * Support conditional expression to evaluate based on the data type for a given field ([#4478](https://github.com/opensearch-project/data-prepper/issues/4478)) |
There was a problem hiding this comment.
Combine this with the PRs:
Suggested change
| * Support conditional expression to evaluate based on the data type for a given field ([#4478](https://github.com/opensearch-project/data-prepper/issues/4478)) | |
| * Support conditional expression to evaluate based on the data type for a given field ([#4478](https://github.com/opensearch-project/data-prepper/issues/4478), [#4523](https://github.com/opensearch-project/data-prepper/pull/4523), [#4500](https://github.com/opensearch-project/data-prepper/pull/4500)) |
| * Checkpoint records at an interval for TPS case when AckSet is enabled ([#4526](https://github.com/opensearch-project/data-prepper/pull/4526)) | ||
| * Complete AcknowledgementSet for checkpoint thread & fix BsonTimeStamp field conversion ([#4525](https://github.com/opensearch-project/data-prepper/pull/4525)) | ||
| * Write stream events that timeout to write to internal buffer in separate thread ([#4524](https://github.com/opensearch-project/data-prepper/pull/4524)) | ||
| * Modify typeof operator grammar and add support for ArrayList ([#4523](https://github.com/opensearch-project/data-prepper/pull/4523)) |
There was a problem hiding this comment.
We can remove:
Suggested change
| * Modify typeof operator grammar and add support for ArrayList ([#4523](https://github.com/opensearch-project/data-prepper/pull/4523)) |
| * Update docdb metrics ([#4508](https://github.com/opensearch-project/data-prepper/pull/4508)) | ||
| * Add BsonType of primary key to metadata ([#4506](https://github.com/opensearch-project/data-prepper/pull/4506)) | ||
| * Add bucket owner support to s3 sink ([#4504](https://github.com/opensearch-project/data-prepper/pull/4504)) | ||
| * Add 'typeof' operator to DataPrepper expressions ([#4500](https://github.com/opensearch-project/data-prepper/pull/4500)) |
There was a problem hiding this comment.
We can remove:
Suggested change
| * Add 'typeof' operator to DataPrepper expressions ([#4500](https://github.com/opensearch-project/data-prepper/pull/4500)) |
dlvenable
requested changes
May 16, 2024
Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>
dlvenable
requested changes
May 20, 2024
Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>
dlvenable
requested changes
May 20, 2024
| @@ -62,7 +59,6 @@ | |||
| * [BUG] Empty DLQ entries when version conflicts occur ([#4301](https://github.com/opensearch-project/data-prepper/issues/4301)) | |||
| * [BUG] otel sources should show a more clear exception when receiving data that cannot be processed based on the configured compression type ([#4022](https://github.com/opensearch-project/data-prepper/issues/4022)) | |||
| * [BUG] : unable to set field_delimiter_regex ([#2946](https://github.com/opensearch-project/data-prepper/issues/2946)) | |||
| * [BUG]Incorrect Behavior of Obfuscate Processor with Predefined Pattern "%{CREDIT_CARD_NUMBER}" [#4340](https://github.com/opensearch-project/data-prepper/issues/4340) | |||
There was a problem hiding this comment.
@dlvenable it is there. I removed a duplicate line.
dlvenable
approved these changes
May 20, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Release Notes for version 2.8
Issues Resolved
Resolves #[Issue number to be closed when this PR is merged]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.