Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support to use old ddb stream image for REMOVE events #4275

Merged
merged 2 commits into from
Mar 19, 2024

Conversation

graytaylor0
Copy link
Member

@graytaylor0 graytaylor0 commented Mar 13, 2024

Description

Adds an option in the to view_on_remove: OLD_IMAGE (defaults to NEW_IMAGE), which will construct an Event from the view type for REMOVE stream events, if old image exists. If it does not exist, the new image will still be used for the Event.

This follows from DynamoDB streams documentation

StreamViewType — Specifies the information that will be written to the stream whenever data in the table is modified:

    KEYS_ONLY — Only the key attributes of the modified item.

    NEW_IMAGE — The entire item, as it appears after it was modified.

    OLD_IMAGE — The entire item, as it appeared before it was modified.

    NEW_AND_OLD_IMAGES — Both the new and the old images of the item

Issues Resolved

Resolves #4261

Check List

  • New functionality includes testing.
  • New functionality has a documentation issue. Please link to it in this PR.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Taylor Gray <tylgry@amazon.com>
@@ -12,8 +12,15 @@ public class StreamConfig {
@JsonProperty(value = "start_position")
private StreamStartPosition startPosition = StreamStartPosition.LATEST;

@JsonProperty("use_old_image_for_deletes")
private boolean useOldImageForDeletes = false;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DynamoDB supports getting stream records as NEW_IMAGE, OLD_IMAGE, KEYS_ONLY, and NEW_AND_OLD_IMAGES. Do you think there may be value for users in using some of these other configurations? For example, if a user wants to perform some processor comparison between the new and old to create a new value.

If so, should we move this from a boolean to an image_view option?

image_view: old_image

https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_streams_StreamRecord.html

We don't have to implement them all now. But, the configuration would be extensible.

Copy link
Member Author

@graytaylor0 graytaylor0 Mar 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There may be room for a more extendible config here, but currently it is required that users have either NEW_AND_OLD_IMAGE or NEW_IMAGE enabled on their streams.

OLD_IMAGE specifically is needed for REMOVE actions, because otherwise an empty item is provided from the stream record.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe something like this mapping makes sense

stream:
   insert_image: new
   modify_image: new
   remove_image: old

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can start with just

stream:
  image_on_remove: old // can be old or new, default to new

This makes the configuration more extendible in the future for something like

stream:
  image_on_insert: new
  image_on_modify: old_and_new
  image_on_remove: old

Signed-off-by: Taylor Gray <tylgry@amazon.com>
@dlvenable dlvenable merged commit d026f56 into opensearch-project:main Mar 19, 2024
67 of 68 checks passed
@graytaylor0 graytaylor0 deleted the OldImageFix branch March 19, 2024 18:06
@kkondaka kkondaka added this to the v2.8 milestone May 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support reading of old image for delete events on DynamoDB source
3 participants