It might be worth seeing if we can leverage the tokio tracing crate for this implementation. It already defines concepts of log levels and components (which it calls targets) and has a rich facility for filtering which logs go where. I realize there might appear to be a circular dependency between their crate and opentelemetry, but I suspect it might be possible to avoid if we (and they) have api vs. sdk crates and our sdk only depends on their api and vice versa.

One option could be extend the existing error_handler for different log levels and so rename it to log_handler.rs. The log level can be configured both at compile time and run-time. I took a dig at this as below:

main...lalitb:opentelemetry-rust:log-handler

So, the existing error handling code:

        if let Some(ref inner) = self.0.inner {
            match inner.lock() {
                Ok(mut locked) => f(&mut locked),
                Err(err) => global::handle_error(err),
            }
        }

to

        if let Some(ref inner) = self.0.inner {
            match inner.lock() {
                Ok(mut locked) => f(&mut locked),
                Err(err) => otel_log_error!(err, Pillar::Trace, Component::Context),
            }
        }

The user can add custom log handler, which can use log, tracing or (e)println!, or anything else.

The macro to log the INFO level log:

    #[macro_export]
    macro_rules! otel_log_info {
        ($message:expr, $pillar:expr, $component:expr) => {
            if is_compile_time_log_level_enabled!(LogLevel::Info) && is_runtime_log_level_enabled(LogLevel::Info) {
                handle_log(Error {
                    level: LogLevel::Info,
                    pillar: $pillar,
                    component: $component,
                    message: $message.to_string(),
                });
            }
        };
    }

Also, as per the @shaun-cox suggestion, we can also directly use the tracing crate for internal logging. The POC for the same :
https://github.com/open-telemetry/opentelemetry-rust/compare/main...lalitb:log-handler-using-tracing?expand=1

• This will cause the infinite logging scenario, so we had to ignore all the internally generated logs at the tracing-appender level.
• Updated the tracing-appender example to showcase it - with a separate layer to process the otel internal logs, while they are ignored by the tracing-appender layer.
• The tracing macros are wrapped under a macros, and called only when the feature flag is enabled. The otel code will only invoke the wrapper, so it allows us to change the tracing crate with (say) log crate in future if we want so.

Also, as per the @shaun-cox suggestion, we can also directly use the tracing crate for internal logging. The POC for the same : https://github.com/open-telemetry/opentelemetry-rust/compare/main...lalitb:log-handler-using-tracing?expand=1

• This will cause the infinite logging scenario, so we had to ignore all the internally generated logs at the tracing-appender level.
• Updated the tracing-appender example to showcase it - with a separate layer to process the otel internal logs, while they are ignored by the tracing-appender layer.
• The tracing macros are wrapped under a macros, and called only when the feature flag is enabled. The otel code will only invoke the wrapper, so it allows us to change the tracing crate with (say) log crate in future if we want so.

Thanks! Can you promote the branch to a PR, so we can start reviewing. I don't see any Circular dep issues, and the infinite logging issue looks trivial to fix either by Otel or by user. We need to develop some conventions about target and eventname that must be followed by OTel components.

The otel code will only invoke the wrapper, so it allows us to change the tracing crate with (say) log crate in future if we want so.

This is a good plan!

Can you promote the branch to a PR, so we can start reviewing.

Have added this - #2128 - for further discussion.

Moving this to 0.27 milestone. This milestone is expected to upgrade several components to RC status, and quality internal logs is a must have for it.