chore(datadog_metrics sink): Re-enable v2 endpoint#24842
Draft
vladimir-dd wants to merge 6 commits intomasterfrom
Draft
chore(datadog_metrics sink): Re-enable v2 endpoint#24842vladimir-dd wants to merge 6 commits intomasterfrom
vladimir-dd wants to merge 6 commits intomasterfrom
Conversation
vladimir-dd
changed the title
pront
reviewed
Mar 4, 2026
Member
pront
left a comment
pront
left a comment
There was a problem hiding this comment.
Leaving some early feedback, this looks good overall.
It would be interesting to run some stress tests, it would be interesting to compare against v1.
There's an existing regression experiment here:
Contributor
Author
thanks, was about to ask for some guidance for performance testing here 🙏 |
pront
reviewed
Mar 4, 2026
vladimir-dd
force-pushed
the
vladimir-dd/metrics-v2
branch
2 times, most recently
from
8148cb3 to
9bb9b2b
Compare
Add regression test to validate datadog_metrics sink v2 endpoint performance under realistic high-throughput DogStatsD load. Test Configuration: - Load: Default lading dogstatsd settings (realistic ~2KB messages) - Throughput: 500 Mb/s → ~250k events/sec - Batch: Default settings (100k max_events, 2s timeout) - Validates batch splitting when payloads exceed v2 size limits This test ensures v2 endpoint correctly handles batch splitting with realistic high-cardinality DogStatsD metrics under load. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
vladimir-dd
force-pushed
the
vladimir-dd/metrics-v2
branch
from
9bb9b2b to
0cb05d2
Compare
Different series endpoints have different uncompressed payload limits (v2 is 12x smaller than v1). This ensures each batch fits in a single HTTP request without splitting, reducing memory overhead. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Member
|
Note: will unsubscribe from this PR until it's "ready for review". When it's ready, we will prioritize reviewing it over other PRs. |
Adds two scripts for benchmarking the Datadog metrics sink v1 vs v2 endpoint performance: - scripts/generate_statsd_load.py: StatsD UDP load generator with configurable rate, metric count, tag cardinality, and high-cardinality tag support. - scripts/benchmark_dd_metrics_v1_v2.py: Orchestrates back-to-back v1/v2 runs, collects throughput/CPU/RSS/HTTP-req metrics from Vector's Prometheus endpoint, and prints a comparison summary. Supports --batch-max-bytes-v1 / --batch-max-bytes-v2 to test with explicit per-endpoint byte limits matching the API uncompressed payload limits (v1: 60 MiB, v2: 5 MiB). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Results from benchmarking the per-endpoint batch size fix against the pre-fix binary at 50k/s and 200k/s with 10 metrics and ~46 tags per event (~932 bytes statsd wire size). Key findings: - Pre-fix, no limits: v2 uses +70% more RSS than v1 (5143 MB vs 3019 MB) - Post-fix, no limits: v2 uses +23% more RSS (fix auto-applies 5 MiB cap) - Post-fix, 200k/s: v2 uses -7% less RSS and -20% less CPU than v1 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… config - Remove references to DatadogMetricsCompression and request_compression from encoder.rs tests (those symbols don't exist in current codebase; they belong to an unmerged compression-options branch) - Fix batcher_user_max_bytes_is_preserved test to avoid struct update syntax with private PhantomData fields in BatchConfig Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
| ## Setup | ||
|
|
||
| - **Branch**: `vladimir-dd/metrics-v2` (includes per-endpoint batch size fix) |
Check failure
Code scanning / check-spelling
Unrecognized Spelling Error
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR switches the Datadog metrics sink to use the v2 series endpoint by default, caps the batcher to fix a memory regression, and adds a regression test.
Memory: v2's smaller payload limit (5 MiB vs 60 MiB for v1) caused a +57–59% RSS regression at 50–100k events/s. Fixed by capping the batcher to the endpoint's payload limit — memory drops to near-parity with v1. Full benchmark results: #24874.
Correctness: End-to-end validated against the real DataDog API (#24879) — all metric types (counter, gauge, set, distribution, aggregated histogram, aggregated summary) pass for both v1 and v2. 36/36 checks pass. v1 and v2 produce identical aggregated values.