Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better metrics for OpenSearch duplicate documents #4343

Closed
dlvenable opened this issue Mar 27, 2024 · 0 comments · Fixed by #4348
Closed

Better metrics for OpenSearch duplicate documents #4343

dlvenable opened this issue Mar 27, 2024 · 0 comments · Fixed by #4348
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@dlvenable
Copy link
Member

Is your feature request related to a problem? Please describe.

We have had some difficulty adding up metrics when some events are duplicates. With the current metrics, we cannot tell how many documents were counted as duplicates in OpenSearch. This can result in being unable to add up the number of events in with the actual documents in OpenSearch.

Goal:

documentsSuccess - documentsDuplicates == count on OpenSearch

Describe the solution you'd like

Add a metrics for duplicate documents. To determine if a document is a duplicate, we can check _seq_no > 0. If this is true, then we can consider this document to be a duplicate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Development

Successfully merging a pull request may close this issue.

1 participant