Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add integration test for Kinesis source #4967

Merged
merged 1 commit into from
Sep 24, 2024

Conversation

sb2k16
Copy link
Member

@sb2k16 sb2k16 commented Sep 20, 2024

Description

This PR is to add an integration test for Kinesis source. As part of the integration test, it will perform the following actions:

  • create a new Kinesis stream in a test account
  • create a lease table for holding the shard specific lease coordination information
  • send a set of records (count = 20) to the stream using putRecord API
  • wait for a period and validate whether source has been able to write 20 records to the buffer.

Issues Resolved

Resolves #[1082]

Check List

  • New functionality includes testing.
  • [X ] Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

chenqi0805
chenqi0805 previously approved these changes Sep 23, 2024

when(pipelineDescription.getPipelineName()).thenReturn(PIPELINE_NAME);

String workerId = "worker-identifier-" + testId;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: final

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @chenqi0805! I will address this.


@BeforeEach
void setup() {
this.acknowledgementSetManager = mock(AcknowledgementSetManager.class);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might consider MockitoAnnotations.openMocks(this) for initializing mocked instances

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @chenqi0805! I will address this.

Signed-off-by: Souvik Bose <souvbose@amazon.com>
Copy link
Collaborator

@oeyh oeyh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

Can you add a readme on how to run the integration tests? Similar to: https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/s3-sink/README.md#developer-guide

}

/**
* CDK stack that creates a Kinesis stream
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this stack supposed to create a Kinesis stream or not? I only see IAM policies being created.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The stream will be created as part of the test. The stack only adds the relevant permissions for accessing Kinesis streams and accessing the dynamoDb table for lease coordination.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do a separate PR for adding the README.

@chenqi0805 chenqi0805 merged commit 54873bf into opensearch-project:main Sep 24, 2024
44 of 48 checks passed
sb2k16 added a commit to sb2k16/data-prepper that referenced this pull request Oct 4, 2024
Signed-off-by: Souvik Bose <souvbose@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants