Add index_types for OTEL logs and metrics #3148

[juergen-walter]

Description

Add index_types for OTEL logs and metrics

Issues Resolved

Resolves #3148

Check List

• New functionality includes testing.
• New functionality has a documentation issue. Please link to it in this PR. --> documentation included in this PR
  • New functionality has javadoc added --> not needed
• Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dlvenable]

@juergen-walter , This is great, thanks for starting this draft! Have you compared this schema with the OpenSearch simple schema for observability?

I also don't see some of the fields from the logs or metrics in here. Are you using dynamic mapping for those?

We recently added support for composable index templates. There is a directly index-template under resources that has those. I meant to move the legacy templates into a new directory in the original PR for that, but didn't. I might go ahead and make that change to help make it clear that there are two sets of templates.

[juergen-walter]

@dlvenable happy to hear your encouraging feedback. I shared early WIP to let you know I started working on it, still a lot to be improved.

I also don't see some of the fields from the logs or metrics in here. Are you using dynamic mapping for those?

Initially I just copied the index templates for trace spans, adjustments for metrics and traces still to be done. For the open source contribution in this PR I would align with the Simple Schema for Observability, thank you for the reminder.

move the legacy templates into a new directory

If you plan prepare or merge changes before this PR has been merged, it would be nice to ping me so I can update the PR

[KarstenSchnitter]

@dlvenable I am not so sure about the Simple Scheme for Observability. We are providing an OpenTelemetry endpoint, that should support generic data. I would expect support for the OpenTelemetry semantic conventions as a whole: https://github.com/open-telemetry/semantic-conventions.

I have run a small PoC, that transfers the YAML configuration from https://github.com/open-telemetry/semantic-conventions/tree/main/model into composable index templates. Each convention maps to one template. Certain assumptions must be made on those mappings. For the three signals traces, metrics and logs a template can be generated covering the base fields. All these templates can than be joined to form a full index pattern to be used. That allows to pick and choose, which conventions should be used. However, the full solution is not yet ready.

[KarstenSchnitter]

Index templates should be improved. Most of the string value fields should be keywords to allow aggregations. And most of them will not have values requiring type "text". Probably all float values should be doubles as most integers should be longs.

[juergen-walter]

Index templates should be improved. Most of the string value fields should be keywords to allow aggregations. And most of them will not have values requiring type "text". Probably all float values should be doubles as most integers should be longs.

I closed all the related conversations and pointed to the Simple Schema for Observability mappings I aligned with. I consider the comments by @KarstenSchnitter to be a valuable feedback/review of the simple schema mappings but I would try to avoid having the respective discussion in this PR.

[juergen-walter]

Hi @dlvenable we would appreciate if we can include this into the next release, so we do not have to implement workarounds.
Can you share your plans on how to proceed?

[dlvenable]

@juergen-walter , It appears that the builds are still failing.

Core integration tests

The core integration tests are failing with:

java.lang.NullPointerException
	at org.opensearch.dataprepper.integration.PipelinesWithAcksIT.two_parallel_pipelines_multiple_records(PipelinesWithAcksIT.java:185)

I'm not sure why this would be failing consistently. Maybe try another rebase for that one?

OpenSearch sink integration tests

For the OpenSearch sink integration tests, you can run these commands to reproduce.

docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" -d opensearchproject/opensearch:2.11.1
sleep 90
./gradlew :data-prepper-plugins:opensearch:integrationTest -Dtests.opensearch.host=localhost:9200 -Dtests.opensearch.user=admin -Dtests.opensearch.password=admin -Dtests.opensearch.bundle=true -Dtests.opensearch.version=opensearch:2.11.1

All versions of OpenSearch are failing.

It is passing on OpenDistro 0.10.0 because this disables the test per the isES6 configuration.

[juergen-walter]

@dlvenable I cannot attend this today due to other obligations. I did not see the failed tests before ... do they appear with some delay or are updated regularly

[dlvenable]

@juergen-walter , These tests are failing consistently.