How to Handle Distributed Tracing with OpenTelemetry Auto-Instrumentation in Java?

[HarshMehta112]

Hi everyone, I'm working on auto-instrumentation of an application using OpenTelemetry in Java, and I'm trying to figure out the best way to handle distributed tracing.

How can I manage distributed tracing effectively with auto-instrumentation in Java using OpenTelemetry?
Is there a way to add extra headers in context propagation without making code changes?
What would be the best approach to ensure that all spans of the distributed trace come together seamlessly?
If anyone has already implemented something similar, would you be willing to share your repository?

Looking forward to hearing your suggestions and any best practices you can share!

[steverao]

Is there a way to add extra headers in context propagation without making code changes?

You can check out https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/examples/extension/README.md#extensions. There are some extension points that you can use.

What would be the best approach to ensure that all spans of the distributed trace come together seamlessly?

Sorry, I don't understand your question, can you elaborate it more?

[HarshMehta112]

Thanks for answer @steverao!

What would be the best approach to ensure that all spans of the distributed trace come together seamlessly?

Currently, I’m working with a distributed microservices architecture. When I make an API request to ServiceA, it internally calls ServiceB. Using OpenTelemetry auto-instrumentation, I generate two separate traces: one for ServiceA and one for ServiceB. Both traces share the same trace ID, which allows me to track the entire request flow across services.

However, I’m facing an issue when storing these traces in the database. The problem is with querying and indexing — Trace ID-based indexing isn’t feasible in my setup, making it difficult to retrieve and correlate traces effectively.

My goal is to ensure that these distributed traces from different services come together seamlessly. I need a solution that enables efficient storage, querying, and retrieval of trace data, which would allow me to easily correlate spans from various services and trace the full request flow.

[laurit]

From your question I get the impression that you are seeking advice on implementing a telemetry backend. This repository is about producing the telemetry that the backends ingest, not about the backend itself. You may have better luck seeking advice elsewhere. I'd suggest you to instead of trying to implement your own backend give a try to one of the existing backends. Since there are multiple open source backends, for example zipkin and jaeger, you can of course examine the source and see how they store and query the data.