Enabling tracing by usage of opentelemetry

[rem1-dev]

Enabling tracing by usage of opentelemetry

[rem1-dev]

@shesek this is same as PR #99 but applied to fresh branch created out of new-index. Commits were squashed.

[rem1-dev]

I'm working on solving those conflict files

[rem1-dev]

@shesek conflicts resolved, it's ready to be merged.

[philippem]

@rem1-dev when testing locally we ran into some errors

tempo-1           | level=warn ts=2024-11-13T20:09:28.400374334Z caller=instance.go:42 msg="TRACE_TOO_LARGE: max size of trace (5000000) exceeded while adding 185917 bytes to trace 6bafc5e34f7227d985e36a2142d95fb6 for tenant single-tenant"
tempo-1           | level=warn ts=2024-11-13T20:09:28.426261953Z caller=instance.go:42 msg="TRACE_TOO_LARGE: max size of trace (5000000) exceeded while adding 185912 bytes to trace 6bafc5e34f7227d985e36a2142d95fb6 for tenant single-tenant"

which may be related to cardinality or size of the spans.

Possible causes:

- Large Trace Data: The trace contains too many spans or overly large data (such as logs or metadata).
- High Cardinality: Excessive number of unique tags, attributes, or data points being attached to spans, leading to large trace size.
- Excessive Span Detail: Individual spans may contain large payloads or detailed information that increase the trace size.
- Improper Sampling Rate: Sampling configuration might be too high, causing too many traces to be captured, some of which may exceed the size limit.
- Large Payloads in Spans: If spans contain large amounts of data (e.g., large JSON bodies, files, or detailed logs), it could push the trace size over the limit.
- Too Many Traces Collected: An overly aggressive tracing configuration that collects an excessive number of traces, increasing the likelihood of exceeding the size limit.
- Long or Complex Transactions: If a trace represents a very large or complex operation (e.g., a multi-step workflow or a distributed operation), it might naturally grow too large.

[rem1-dev]

@philippem how can I reproduce this locally?

[philippem]

@shesek what do you think about adding a new macro to reduce the amount of copy-paste? will require new module

[rem1-dev]

@shesek what do you think about adding a new macro to reduce the amount of copy-paste? will require new module

@philippem: I think that wrapper macro actually requires too much code restructuring:

• in order for it to be used as simply as:

#[instrumented]
    pub fn request(&self, method: &str, params: Value) -> Result<Value> { ... }

instead of:

    pub fn request(&self, method: &str, params: Value) -> Result<Value> {
    instrumented!(
    ...
    );
    }

it would require defining procedural macro. But that would require separate crate and I'll have to change code structure from:

electrs_crate/
   ├── Cargo.toml
   └── src/
       └── electrum/
       └── elements/
       └── new_index/
       └── lib.rs
       └── metrics.rs
       └── daemon.rs
       └── *.rs

to:

electrs/
├── Cargo.toml          # (Workspace root)
├── electrs_crate/
   ├── Cargo.toml
   └── src/
       └── electrum/
       └── elements/
       └── new_index/
       └── lib.rs
       └── metrics.rs
       └── daemon.rs
       └── *.rs
└── macros_crate/
    ├── Cargo.toml
    └── src/
        └── lib.rs