Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add preview_datadog_agent_sampling #6112

Merged
merged 36 commits into from
Dec 16, 2024
Merged
Changes from 1 commit
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
a1b638d
Add `experimental_datadog_agent_sampling`
Oct 3, 2024
001af54
Merge branch 'dev' into bryn/datadog-agent-support
BrynCooke Oct 3, 2024
42247b7
Update docs
Oct 4, 2024
7ad4d02
Update docs
Oct 4, 2024
d8bf352
Update docs
Oct 4, 2024
110917c
Update docs
Oct 4, 2024
562b7ac
Split the changelogs
Oct 4, 2024
1776e59
Remove commented out code for auto enabling sampling
Oct 4, 2024
5d1b0b4
Merge branch 'dev' into bryn/datadog-agent-support
BrynCooke Oct 15, 2024
9b60c38
Update docs to indicate that preview datadog agent sampling will NOT …
Oct 16, 2024
16cd222
Update docs/source/configuration/telemetry/exporters/tracing/datadog.mdx
BrynCooke Oct 25, 2024
d7532ce
Update docs/source/configuration/telemetry/exporters/tracing/datadog.mdx
BrynCooke Oct 25, 2024
c91a233
Apply suggestions from code review
BrynCooke Oct 25, 2024
98624c7
Apply suggestions from code review
BrynCooke Oct 25, 2024
067eb56
Merge branch 'dev' into bryn/datadog-agent-support
BrynCooke Oct 25, 2024
43aa702
Merge dev
Nov 4, 2024
17e88fa
Merge remote-tracking branch 'public/dev' into bryn/datadog-agent-sup…
abernix Nov 14, 2024
8726737
Merge branch 'dev' into bryn/datadog-agent-support
BrynCooke Nov 19, 2024
8909929
Merge branch 'dev' into bryn/datadog-agent-support
BrynCooke Nov 21, 2024
1105d31
Merge branch 'dev' into bryn/datadog-agent-support
BrynCooke Dec 6, 2024
30ffb13
Improve datadog and otlp sampling tests
Dec 10, 2024
b8a8e20
Merge branch 'dev' into bryn/datadog-agent-support
BrynCooke Dec 10, 2024
4490216
Lint and test fixes
Dec 10, 2024
bfeca6e
Improve integration tester
Dec 13, 2024
530eecd
Fix coprocessor test
Dec 13, 2024
0a1e1f7
Fix redis test
Dec 13, 2024
e81b5ec
Merge dev
Dec 13, 2024
14c3ed6
Lint
Dec 13, 2024
1c0a232
Lint
Dec 13, 2024
437f053
Fix a couple of new tests from merge
Dec 13, 2024
e701fda
Make test less flaky
Dec 13, 2024
be68420
Revert change to propagator order
Dec 13, 2024
1e6b559
Add test for custom propagation overriding the DD propagator
Dec 13, 2024
55ec86e
Add test and fix datadog propagator to fall back to the span sampling…
Dec 13, 2024
3b5d256
Add test for non datadog agent propagation zero percent sampling.
Dec 13, 2024
0632ee6
Improve trace_id test
Dec 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update docs
  • Loading branch information
bryn committed Oct 4, 2024
commit d8bf3528c00d56819b9ad4fadbdba5686564f9c2
Original file line number Diff line number Diff line change
@@ -313,12 +313,12 @@ telemetry:
## Sampler configuration

When using Datadog you will need to take into consideration if you want to use the Datadog APM view or rely on OTLP metrics to gain insight into the Router's performance.
The Datadog APM vie is driven by traces, and for this to be accurate 100% of requests must be sampled and sent to the Datadog agent.
The Datadog APM view is driven by traces, and for this to be accurate 100% of requests must be sampled and sent to the Datadog agent.

Tracing is expensive both in terms of APM costs but also Router performance, and typically you will want to set the `sampler` to a low value ion production environments.
However, this will mean that the APM view will only show a small percentage of traces.

Datadog agent sampling is a mode where ALL traces are sent to the Datadog agent, but only a percentage of them are forwarded to Datadog. This makes the APM view accurate while keeping costs low
Datadog Agent sampling is a mode where ALL traces are sent to the Datadog agent, but only a percentage of them are forwarded to Datadog. This makes the APM view accurate while keeping costs low
at the cost of the Router having an effective sample rate of 100% under the hood.

Here are some guides on how to configure the `sampler` and `preview_datadog_agent_sampling` to get the desired behavior:
@@ -332,7 +332,8 @@ telemetry:
exporters:
tracing:
common:
# Only 10 percent of spans will be forwarded from the Datadog agent to Datadog. Experiment to find a value that is good for you!
# All requests will be traced and sent to the Datadog agent.
# Only 10 percent of spans will be forwarded from the Datadog agent to Datadog.
preview_datadog_agent_sampling: true
sampler: 0.1
```
@@ -347,6 +348,7 @@ telemetry:
exporters:
tracing:
common:
# All requests will be traced and sent to the Datadog agent.
sampler: always_on
```

@@ -355,6 +357,7 @@ Datadog agent config:
otlp_config:
traces:
probabilistic_sampling:
# Only 10 percent of spans will be forwarded to Datadog
sampling_percentage: 10
```

@@ -367,6 +370,7 @@ telemetry:
exporters:
tracing:
common:
# Only 10 percent of requests will be traced and sent to the Datadog agent. The APM view will only show a subset of total request data but the Router will perform better.
sampler: 0.1
preview_datadog_agent_sampling: false
```
Loading