Skip to content

Latest commit

 

History

History
351 lines (287 loc) · 16.8 KB

README.md

File metadata and controls

351 lines (287 loc) · 16.8 KB
Raw

Extending the OpenTelemetry .NET SDK

Exporter

OpenTelemetry .NET SDK has provided the following built-in trace exporters:

Custom exporters can be implemented to send telemetry data to places which are not covered by the built-in exporters:

  • Exporters should derive from OpenTelemetry.BaseExporter<Activity> (which belongs to the OpenTelemetry package) and implement the Export method.
  • Exporters can optionally implement the OnForceFlush and OnShutdown method.
  • Depending on user's choice and load on the application, Export may get called with one or more activities.
  • Exporters will only receive sampled-in and ended activities.
  • Exporters should not throw exceptions from Export, OnForceFlush and OnShutdown.
  • Exporters should not modify activities they receive (the same activity may be exported again by different exporter).
  • Exporters are responsible for any retry logic needed by the scenario. The SDK does not implement any retry logic.
  • Exporters should avoid generating telemetry and causing live-loop, this can be done via OpenTelemetry.SuppressInstrumentationScope.
  • Exporters should use Activity.TagObjects collection instead of Activity.Tags to obtain the full set of attributes (tags).
  • Exporters should use ParentProvider.GetResource() to get the Resource associated with the provider.
class MyExporter : BaseExporter<Activity>
{
    public override ExportResult Export(in Batch<Activity> batch)
    {
        using var scope = SuppressInstrumentationScope.Begin();

        foreach (var activity in batch)
        {
            Console.WriteLine($"Export: {activity.DisplayName}");
        }

        return ExportResult.Success;
    }
}

A demo exporter which simply writes activity name to the console is shown here.

Apart from the exporter itself, you should also provide extension methods as shown here. This allows users to add the Exporter to the TracerProvider as shown in the example here.

Exporting Activity Status

DiagnosticSource package did not originally have a dedicated field for storing Status, and hence, users were encouraged to follow the convention of storing status using tags "otel.status_code" and "otel.status_description". DiagnosticSource version 6.0.0 added Status and StatusDescription to Activity class. Exporters which support reading status from Activity directly should fall back to retrieving status from the tags described above, to maintain backward compatibility. ConsoleActivityExporter may be used as a reference.

Instrumentation Library

The inspiration of the OpenTelemetry project is to make every library observable out of the box by having them call OpenTelemetry API directly. However, many libraries will not have such integration, and as such there is a need for a separate library which would inject such calls, using mechanisms such as wrapping interfaces, subscribing to library-specific callbacks, or translating existing telemetry into OpenTelemetry model.

A library which enables instrumentation for another library is called Instrumentation Library and the library it instruments is called the Instrumented Library. If a given library has built-in instrumentation with OpenTelemetry, then instrumented library and instrumentation library will be the same.

The OpenTelemetry .NET Github repo ships the following instrumentation libraries. The individual docs for them describes the library they instrument, and steps for enabling them.

More community contributed instrumentations are available in OpenTelemetry .NET Contrib. If you are writing an instrumentation library yourself, use the following guidelines.

Writing own instrumentation library

This section describes the steps required to write your own instrumentation library.

If you are writing a new library or modifying an existing library, the recommendation is to use ActivitySource API/OpenTelemetry API to instrument it and emit activity/span. If a library is instrumented using ActivitySource API, then there is no need of writing a separate instrumentation library, as instrumented and instrumentation library become same in this case. For applications to collect traces from this library, all that is needed is to enable the ActivitySource for the library using AddSource method of the TracerProviderBuilder. The following section is applicable only if you are writing an instrumentation library for an instrumented library which you cannot modify to emit activities directly.

Writing an instrumentation library typically involves 3 steps.

  1. First step involves "hijacking" into the target library. The exact mechanism of this depends on the target library itself. For example, StackExchangeRedis library allows hooks into the library, and the StackExchangeRedis instrumentation library in this case, leverages them. Another example is System.Data.SqlClient for .NET Framework, which publishes events using EventSource. The SqlClient instrumentation library, in this case subscribes to the EventSource callbacks.

  2. Second step is to emit activities using the ActivitySource API. In this step, the instrumentation library emits activities on behalf of the target instrumented library. Irrespective of the actual mechanism used in first step, this should be uniform across all instrumentation libraries. The ActivitySource must be created using the name and version of the instrumentation library (eg: "OpenTelemetry.Instrumentation.StackExchangeRedis") and not the instrumented library (eg: "StackExchange.Redis")

    1. Context Propagation: If your library initiates out of process requests or accepts them, the library needs to inject the PropagationContext to outgoing requests and extract the context and hydrate the Activity/Baggage upon receiving incoming requests. This is only required if you're using your own protocol to communicate over the wire. (i.e. If you're using an already instrumented HttpClient or GrpcClient, this is already provided to you and do not require injecting/extracting PropagationContext explicitly again.)

  3. Third step is an optional step, and involves providing extension methods on TracerProviderBuilder, to enable the instrumentation. This is optional, and the below guidance must be followed:

    1. If the instrumentation library requires state management tied to that of TracerProvider, then it must register itself with the provider with the AddInstrumentation method on the TracerProviderBuilder. This causes the instrumentation to be created and disposed along with TracerProvider. If the above is required, then it must provide an extension method on TracerProviderBuilder. Inside this extension method, it should call the AddInstrumentation method, and AddSource method to enable its ActivitySource for the provider. An example instrumentation using this approach is StackExchangeRedis instrumentation

    2. If the instrumentation library does not requires any state management tied to that of TracerProvider, then providing TracerProviderBuilder extension method is optional. If provided, then it must call AddSource to enable its ActivitySource for the provider.

    3. If instrumentation library does not require state management, and is not providing extension method, then the name of the ActivitySource used by the instrumented library must be documented so that end users can enable it using AddSource method on TracerProviderBuilder.

Special case : Instrumentation for libraries producing legacy Activity

There is a special case for libraries which are already instrumented to produce Activity, but using the DiagnosticSource method. These are referred to as "legacy Activity" in this repo. These libraries already create activities but they do so by using the Activity constructor directly, rather than using ActivitySource.StartActivity method. These activities does not by default runs through the sampler, and will have their Kind set to internal and they'll have empty ActivitySource name associated with it.

Some common examples of such libraries include ASP.NET, ASP.NET Core, HTTP client .NET Core . Instrumentation libraries for these are already provided in this repo. The OpenTelemetry .NET Contrib repository also has instrumentations for libraries like ElasticSearchClient etc. which fall in this category.

If you are writing instrumentation for such library, it is recommended to refer to one of the above as a reference.

Processor

OpenTelemetry .NET SDK has provided the following built-in processors:

Custom processors can be implemented to cover more scenarios:

  • Processors should inherit from OpenTelemetry.BaseProcessor<Activity> (which belongs to the OpenTelemetry package), and implement the OnStart and OnEnd methods.
  • Processors can optionally implement the OnForceFlush and OnShutdown methods. OnForceFlush should be thread safe.
  • Processors should not throw exceptions from OnStart, OnEnd, OnForceFlush and OnShutdown.
  • OnStart and OnEnd should be thread safe, and should not block or take long time, since they will be called on critical code path.
class MyProcessor : BaseProcessor<Activity>
{
    public override void OnStart(Activity activity)
    {
        Console.WriteLine($"OnStart: {activity.DisplayName}");
    }

    public override void OnEnd(Activity activity)
    {
        Console.WriteLine($"OnEnd: {activity.DisplayName}");
    }
}

A demo processor is shown here.

Enriching Processor

A common use case of writing custom processor is to enrich activities with additional tags. An example of such an "EnrichingProcessor" is shown here. Such processors must be added before the exporters.

This processor also shows how to enrich Activity with additional tags from the Baggage.

Many instrumentation libraries shipped from this repo provides a built-in Enrich option, which may also be used to enrich activities. Instrumentation library provided approach may offer additional capabilities such as offering easy access to more context (library specific).

Filtering Processor

Another common use case of writing custom processor is to filter Activities from being exported. Such a "FilteringProcessor" can be written as a wrapper around an underlying processor. An example "FilteringProcessor" is shown here.

When using such a filtering processor, instead of using extension method to register the exporter, they must be registered manually as shown below:

using var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .SetSampler(new MySampler())
    .AddSource("OTel.Demo")
    .AddProcessor(new MyFilteringProcessor(
        new SimpleActivityExportProcessor(new MyExporter("ExporterX")),
        (act) => true))
    .Build();

Most instrumentation libraries shipped from this repo provides a built-in Filter option to achieve the same effect. In such cases, it is recommended to use that option as it offers higher performance.

Sampler

OpenTelemetry .NET SDK has provided the following built-in samplers:

Custom samplers can be implemented to cover more scenarios:

  • Samplers should inherit from OpenTelemetry.Trace.Sampler (which belongs to the OpenTelemetry package), and implement the ShouldSample method.
  • ShouldSample should be thread safe, and should not block or take long time, since it will be called on critical code path.
class MySampler : Sampler
{
    public override SamplingResult ShouldSample(in SamplingParameters samplingParameters)
    {
        return new SamplingResult(SamplingDecision.RecordAndSampled);
    }
}

A demo sampler is shown here.

Resource Detector

OpenTelemetry .NET SDK provides a resource detector for detecting resource information from the OTEL_RESOURCE_ATTRIBUTES and OTEL_SERVICE_NAME environment variables.

Custom resource detectors can be implemented:

  • ResourceDetectors should inherit from OpenTelemetry.Resources.IResourceDetector, (which belongs to the OpenTelemetry package), and implement the Detect method.

A demo ResourceDetector is shown here.

References