Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide an OpenSearch API source #4180

Open
1 of 2 tasks
dlvenable opened this issue Feb 23, 2024 · 2 comments
Open
1 of 2 tasks

Provide an OpenSearch API source #4180

dlvenable opened this issue Feb 23, 2024 · 2 comments
Labels
enhancement New feature or request Roadmap:Observability/Log Analytics Project-wide roadmap label

Comments

@dlvenable
Copy link
Member

dlvenable commented Feb 23, 2024

Is your feature request related to a problem? Please describe.

I'd like to be able to point existing OpenSearch clients that perform ingestion to Data Prepper instead.

Describe the solution you'd like

Create a new source - opensearch_api which supports the Document APIs that perform ingestion.

Minimally, this can be:

source:
  opensearch_api:

Some more options:

source:
  opensearch_api:
    port: 9200
    ssl: true
    ssl_certificate_file: /path/to/certificate
    ssl_key_file: /path/to/private.key
    thread_count: 200
    path_prefix: opensearch/
    authentication:
      http_basic:
        username: ingest_client
        password: ${{aws_secrets:ingest:password}}

Additional context

This can start with only the _bulk API and this could supersede #248. Over time we can add the other endpoints as well.

This would be available for _bulk using the path:

https://localhost:9200/_bulk

Tasks

@dlvenable
Copy link
Member Author

We may also want to make the REST APIs configurable so that users can enable only some.

For example.

source:
  opensearch_api:
    apis: [bulk, index, update, delete]

@dlvenable
Copy link
Member Author

Ideally, we can combine much of this code with the http source. It may be good to add a new project to Data Prepper for http-server-common which supports common HTTP code. Much of the code can be similar between the opensearch_api source and the http source.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Roadmap:Observability/Log Analytics Project-wide roadmap label
Projects
Status: New
Development

No branches or pull requests

2 participants