-
Notifications
You must be signed in to change notification settings - Fork 190
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Remove validation that made keys starting or ending with . - or _ inv…
…alid, catch all exceptions in the parse json processor Signed-off-by: Taylor Gray <tylgry@amazon.com>
- Loading branch information
1 parent
1dd8bd3
commit f91ead4
Showing
5 changed files
with
100 additions
and
40 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,69 @@ | ||
# OpenSearch Source | ||
|
||
This is the Date Prepper OpenSearch source plugin that processes indices either OpenSearch, Elasticsearch, | ||
or Amazon OpenSearch Service. It is meant for migrating index data from a cluster. | ||
|
||
Note: Only fully tested versions with be listed below. It is likely many more versions are supported already, but it is untested. | ||
|
||
The OpenSearch source is compatible with the following OpenSearch versions: | ||
* 2.5 | ||
|
||
And is compatible with the following Elasticsearch versions: | ||
* 7.10 | ||
|
||
# Usages | ||
|
||
### Amazon OpenSearch Service | ||
|
||
The OpenSearch sink can also be configured for an Amazon OpenSearch Service domain. See [security](security.md) for details. | ||
|
||
```yaml | ||
opensearch-source-pipeline: | ||
source: | ||
opensearch: | ||
connection: | ||
insecure: true | ||
hosts: [ "https://search-my-domain-soopywaovobopgs8ywurr3utsu.us-east-1.es.amazonaws.com" ] | ||
aws: | ||
region: "us-east-1" | ||
sts_role_arn: "arn:aws:iam::123456789012:role/my-domain-role" | ||
``` | ||
## Configuration | ||
- `hosts`: A list of IP addresses of OpenSearch or Elasticsearch nodes. | ||
|
||
|
||
- `username`: | ||
|
||
|
||
- `password`: | ||
|
||
|
||
- `aws` (Optional) : AWS configurations. See [AWS Configuration](#aws_configuration) for details. SigV4 is enabled by default when this option is used. | ||
|
||
|
||
- `search_options` (Optional) : See [Search Configuration](#search_configuration) for details | ||
|
||
|
||
- `indices` (Optional): See [Indices Configurations](#indices_configuration) for filtering options. | ||
|
||
|
||
- `scheduling` (Optional): See [Scheduling Configuration](#scheduling_configuration) for details | ||
|
||
|
||
- `connection` (Optional): See [] | ||
|
||
### <a name="aws_configuration">AWS Configuration</a> | ||
|
||
* `region` (Optional) : The AWS region to use for credentials. Defaults to [standard SDK behavior to determine the region](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/region-selection.html). | ||
* `sts_role_arn` (Optional) : The STS role to assume for requests to AWS. Defaults to null, which will use the [standard SDK behavior for credentials](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html). | ||
* `sts_header_overrides` (Optional): A map of header overrides to make when assuming the IAM role for the source plugin. | ||
|
||
### <a name="search_configuration">Search Configuration</a> | ||
|
||
### <a name="scheduling_configuration">Scheduling Configuration</a> | ||
|
||
### <a name="connection_configuration">Connection Configuration</a> | ||
|
||
### <a name="indices_configuration">Indices Configuration</a> |
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters