[exporter/elasticsearch] Support both v7 and v8 elasticsearch clients and default to v8

[carsonip]

Component(s)

exporter/elasticsearch

Is your feature request related to a problem? Please describe.

Currently elasticsearch exporter uses go-elasticsearch v7 because of the "forward compatibility" described in https://github.com/elastic/go-elasticsearch?tab=readme-ov-file#elasticsearch and raised in #16310. However, this means that v8 stack users are also forced to use go-elasticsearch v7 as mentioned in #29689. v8 go-elasticsearch client gets the bugfixes and features that may not be available in v7 go-elasticsearch client. Elasticsearch exporter should support both v7 and v8 elasticsearch client, and ideally default to v8 (which may be a breaking change).

Describe the solution you'd like

• Support both v7 and v8 go-elasticsearch client and expose it as a configuration option.
• Default to v8 client

Describe alternatives you've considered

No response

Additional context

Related to #32378

[github-actions]

Pinging code owners:

• exporter/elasticsearch: @JaredTan95 @ycombinator

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[JaredTan95]

Agree, this upgrade has been mentioned many times and I think it is a necessary upgrade

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/elasticsearch: @JaredTan95 @ycombinator @carsonip

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[carsonip]

Still relevant, and this should be ready to implement as elasticsearchexporter is now using go-docappender

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/elasticsearch: @JaredTan95 @ycombinator @carsonip

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[carsonip]

Still relevant as mentioned above.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/elasticsearch: @JaredTan95 @carsonip @lahsivjar

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[lahsivjar]

/label -Stale

[TylerHelmuth]

@andrzej-stencel @ChrsMark any progress on this?

[lahsivjar]

@TylerHelmuth We haven't started working on this yet so no progress so far.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/elasticsearch: @JaredTan95 @carsonip @lahsivjar

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[lahsivjar]

/label -Stale

[axw]

I was looking into updating to go-elasticsearch/v8 again. go-elasticsearch/v8 can be used with 7.17.x, but not older ES versions, as shown in #16310.

Given that 7.x versions of Elasticsearch prior to 7.17.x are unsupported by Elastic, is it necessary to continue support for them in the exporter? If users wish to send to an old, unsupported version of ES, they could pin to an older version of OTel Collector too.

[carsonip]

Given that 7.x versions of Elasticsearch prior to 7.17.x are unsupported by Elastic, is it necessary to continue support for them in the exporter? If users wish to send to an old, unsupported version of ES, they could pin to an older version of OTel Collector too.

Sounds fine to me.

On a related note, I just found out that an issue with OTel mapping mode with < 8.12: #37283 . There is no problem compiling and running the code, as es exporter now uses v8 esapi on a v7 client, which is fine for most part until we start using params supported in a newer v8. This is also the reason behind "forward compatibility" in go-elasticsearch. Using a v8.12 go-elasticsearch that supports require_data_stream will cause the request to fail when sent to 8.11 ES.

I imagine we'll have to actively work around these compat issues in the es exporter, then I wonder if #37187 version config should not only specify major, but also minor.

cc @felixbarny

[rogercoll]

Maybe for a separate issue, but I was wondering if we could unify the code of creating Elasticseach clients into an internal shared package. That way, any component that connects to Elasticsearch could reuse that code. Note that at the moment the Elasticsearch receiver does not use the go-elasticsearch package to create the client https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/elasticsearchreceiver/client.go#L65 (also, I am working in a couple of components that would benefit from that)

[axw]

After discussing options with code owners, I've opened #37333 for review.

@rogercoll I think having a common package for Elasticsearch client config sounds sensible.