Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve OpenSearch bulk retry logging #4268

Open
travisbenedict opened this issue Mar 12, 2024 · 1 comment
Open

Improve OpenSearch bulk retry logging #4268

travisbenedict opened this issue Mar 12, 2024 · 1 comment
Labels
plugin - sink A plugin to write data to a destination.

Comments

@travisbenedict
Copy link
Contributor

travisbenedict commented Mar 12, 2024

Is your feature request related to a problem? Please describe.

It's difficult to track bulk retry failures if Data Prepper is writting to multiple indexes.

Describe the solution you'd like

  1. Log the bulk failure the first time that there is a failure instead of only doing it every 5 times.

  2. Update logging to show the OpenSearch index for which the bulk request failed.

if (retryCount % 5 == 0) {
LOG.warn("Bulk Operation Failed. Number of retries {}. Retrying... ", retryCount, exceptionFromRequest);

Describe alternatives you've considered (Optional)
N/A

Additional context
N/A

@dlvenable
Copy link
Member

We can work this along with #4195 to greatly improve our error logging.

@dlvenable dlvenable added plugin - sink A plugin to write data to a destination. and removed untriaged labels Mar 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
plugin - sink A plugin to write data to a destination.
Projects
Development

No branches or pull requests

2 participants