Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] OpenSearch source plugin requires auth configuration even when the source cluster does not need any #2939

Closed
kartg opened this issue Jun 26, 2023 · 0 comments · Fixed by #2942
Assignees
Labels
bug Something isn't working

Comments

@kartg
Copy link
Member

kartg commented Jun 26, 2023

Describe the bug
Firstly, thanks for merging in support for the scroll API for the ElasticSearch accessor! When I attempted to use this plugin with a test ElasticSearch OSS source cluster, I ran into the following error:

....
2023-06-26T22:30:19,634 [historical-data-migration-sink-worker-2-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [historical-data-migration] Sink is ready, starting source...
2023-06-26T22:30:19,641 [historical-data-migration-sink-worker-2-thread-1] INFO  org.opensearch.dataprepper.plugins.source.opensearch.worker.client.OpenSearchClientFactory - Using username and password for auth for the OpenSearch source
2023-06-26T22:30:19,642 [historical-data-migration-sink-worker-2-thread-1] ERROR org.opensearch.dataprepper.pipeline.common.PipelineThreadPoolExecutor - Pipeline [historical-data-migration] process worker encountered a fatal exception, cannot proceed further
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Username may not be null
	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]
	at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:?]
	at org.opensearch.dataprepper.pipeline.common.PipelineThreadPoolExecutor.afterExecute(PipelineThreadPoolExecutor.java:70) [data-prepper-core-2.4.0-SNAPSHOT.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1137) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.IllegalArgumentException: Username may not be null
	at org.apache.http.util.Args.notNull(Args.java:54) ~[httpcore-4.4.16.jar:4.4.16]
	at org.apache.http.auth.UsernamePasswordCredentials.<init>(UsernamePasswordCredentials.java:81) ~[httpclient-4.5.14.jar:4.5.14]
	at org.opensearch.dataprepper.plugins.source.opensearch.worker.client.OpenSearchClientFactory.attachUsernamePassword(OpenSearchClientFactory.java:175) ~[opensearch-source-2.4.0-SNAPSHOT.jar:?]
	at org.opensearch.dataprepper.plugins.source.opensearch.worker.client.OpenSearchClientFactory.createOpenSearchRestClient(OpenSearchClientFactory.java:137) ~[opensearch-source-2.4.0-SNAPSHOT.jar:?]
	at org.opensearch.dataprepper.plugins.source.opensearch.worker.client.OpenSearchClientFactory.provideOpenSearchClient(OpenSearchClientFactory.java:75) ~[opensearch-source-2.4.0-SNAPSHOT.jar:?]
	at org.opensearch.dataprepper.plugins.source.opensearch.worker.client.SearchAccessorStrategy.getSearchAccessor(SearchAccessorStrategy.java:61) ~[opensearch-source-2.4.0-SNAPSHOT.jar:?]
	at org.opensearch.dataprepper.plugins.source.opensearch.OpenSearchSource.startProcess(OpenSearchSource.java:49) ~[opensearch-source-2.4.0-SNAPSHOT.jar:?]
	at org.opensearch.dataprepper.plugins.source.opensearch.OpenSearchSource.start(OpenSearchSource.java:41) ~[opensearch-source-2.4.0-SNAPSHOT.jar:?]
	at org.opensearch.dataprepper.pipeline.Pipeline.startSourceAndProcessors(Pipeline.java:210) ~[data-prepper-core-2.4.0-SNAPSHOT.jar:?]
	at org.opensearch.dataprepper.pipeline.Pipeline.lambda$execute$2(Pipeline.java:251) ~[data-prepper-core-2.4.0-SNAPSHOT.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	... 2 more
2023-06-26T22:30:19,654 [historical-data-migration-sink-worker-2-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [historical-data-migration] - Received shutdown signal with processor shutdown timeout PT30S and sink shutdown timeout PT30S. Initiating the shutdown process
...

It looks like the plugin is expecting username and password configurations in the pipeline even when the source cluster does not have security enabled (which mine does not since i'm using the OSS distro and security is an X-Pack plugin)

To Reproduce
Steps to reproduce the behavior:

  1. Start/use an ElasticSearch OSS source cluster (i'm using ES OSS 7.10.2) without any auth
  2. Configure a Data Prepper pipeline with the following source:
test-pipeline:
    source:
        opensearch:
            hosts: ["<es-oss-host>"]
  1. Start Data Prepper using this pipeline and observe the error

Expected behavior
The above source configuration should work without errors

Screenshots
N/A, stack-trace provided above

Environment (please complete the following information):

  • OS: MacOS 12.6.6
  • Version: Data Prepper 2.4.0-SNAPSHOT

Additional context
I am able to work around this problem by providing dummy values for username and password in the config:

test-pipeline:
    source:
        opensearch:
            hosts: ["<es-oss-host>"]
            username: "dummy"
            password: "dummy"

The ES source cluster ignores/does not parse these values so the API call goes through.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

2 participants