Add support for APM Remote Configuration

[dmehala]

Description

Remote Configuration allows to remotely configure the library. For now, it is only possible to update the trace sampling rate remotely.

How to test

An API Key with Remote Configuration enabled is required.

1. Run the hasher example.
2. Generate some traces so the service can appear in the Service Catalogue.
3. In Datadog's UI go to the Service Catalogue and select the dd-trace-cpp-example service.
4. An APM Instrumentation banner should be visible.

[dgoffredo]

I'm playing with my own branch, making suggestions for this PR.

In looking at TracerId and the related parts of the remote config API, I noticed the following paragraph from our internal "[RFC] Integrating with Remote Config in a Tracer":

The Client.ClientState field in ClientGetConfigsRequest MUST contain the global state of RC in the tracer. There MUST NOT exist separate instances of RC clients in a tracer making separate ClientGetConfigsRequest with their own separated Client.ClientState. We use this state to track individual tracers in the agent and the backend. Having multiple components in the same tracer each call this API with separated state will duplicate a lot of data and cause issues in the attribution of configurations.

What is meant by "separate" here? My guess is that each Client.ClientState is specific to a unique tracer. What then determines the identify of a "tracer"? Is it just client.id? Is it client.id combined with client.client_tracer.*, as you have largely done in defining struct TracerId?

The notion of "tracer ID" is suspect in my mind. How does it all fit together?

[dmehala]

What is meant by "separate" here? My guess is that each Client.ClientState is specific to a unique tracer. What then determines the identify of a "tracer"? Is it just client.id? Is it client.id combined with client.client_tracer.*, as you have largely done in defining struct TracerId?

I asked RC people, here are there answer:

client.id is used to identify individual tracers
There is also assumptions being made such as "it's impossible to have two different client.id for a single client.runtimeID". Breaking those assumptions will likely break some UIs and potentially confuse systems that distribute configurations to specific tracers

From my understanding it is a combination client.id and client.client_tracer.runtime_id. If I remember they should not be the same but one client.id is associated to one client.client_tracer.runtime_id. Did I missed something?

The notion of "tracer ID" is suspect in my mind. How does it all fit together?

My main objective in incorporating the TracerID data structure was to enhance code simplicity and improve expressiveness.

[dgoffredo]

There is also assumptions being made such as "it's impossible to have two different client.id for a single client.runtimeID".

Currently, each class Tracer creates its own RuntimeID. When envoy or nginx creates worker threads/processes, each gets its own Tracer, and so each gets its own RuntimeID.

@cgilmour and I decided that it would be better if these workers shared the same RuntimeID. They are, after all, part of the same runtime. The first bit of code that would enforce this are the proposed fixes to our envoy code that enable newer versions of dd-trace-cpp. See DatadogTracerFactory::makeConfig in that change set.

I'm concerned that now each worker will have the same "tracer ID," (same runtime, env, service) but that the workers' client.id will all be different. client.id is generated once per Tracer.

What I'm trying to get at, asking these questions, is what is a tracer? Worst case scenario, there is no clear definition.

Here's my opinion: Workers within a proxy are the "same tracer." They're part of the same runtime and share the same configuration. So, I figure each of these workers should appear identical to the remote config backend. Will that be a problem? I think it will be a problem.

Alternatively, workers can share a runtime ID, env, and service, but have different client IDs. What is meant by "client"? What is the relationship between "client" and "tracer"?

Let's iron out these definitions before committing the code. At the very least, it will allow us to write good documentation for the TracerID component.

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-01-09 15:58:19

Comparing candidate commit 99c1c65 in PR branch dmehala/remote-config-impl with baseline commit cc703b2 in branch main.

Found 1 performance improvements and 0 performance regressions! Performance is the same for 0 metrics, 0 unstable metrics.

scenario:BM_TraceTinyCCSource

• 🟩 execution_time [-12.009ms; -11.938ms] or [-14.495%; -14.409%]

[dgoffredo]

Looking good. Please see my suggestions in #83.

Once we've discussed that, I'll mark this approved.

[dmehala]

@dgoffredo I added your changes in 8a9d59b and 3444245

[dgoffredo]

Very nice! This feature came out well.

Once this is merged, I'll see whether it breaks Envoy's unit tests. We can keep the two concerns decoupled.