Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[statsdreceiver] add source attributes and option to disable aggregation by source address #33010

Closed
wants to merge 2 commits into from
Closed

[statsdreceiver] add source attributes and option to disable aggregation by source address #33010

wants to merge 2 commits into from

Conversation

Manuelraa
Copy link

Description:

Fixes #23809

Link to tracking Issue: #23809

Testing:
Added unit tests for the new attributes for both setting states.

Tested by running collector and manually pushing metrics.

# Test same instance two connections
echo "test.metric:1|ms|@1|#myKey:foo" | nc -v -u 127.0.0.1 8125;
echo "test.metric:2|ms|@1|#myKey:foo" | nc -v -u 127.0.0.1 8125;

# Test same instance one connection
echo -e "test.metric:1|ms|@1|#myKey:bar\ntest.metric:2|ms|@1|#myKey:bar" | nc -v -u 127.0.0.1 8125;
Detaching and terminating target process
dlv dap (910189) exited with code: 0
Starting: /home/user/go/bin/dlv dap --listen=127.0.0.1:39177 --log-dest=3 from /home/user/sources/public/opentelemetry-collector-contrib/cmd/oteltestbedcol
DAP server listening at: 127.0.0.1:39177
Type 'dlv help' for list of commands.
2024-05-13T11:23:49.017+0200	info	service@v0.100.0/service.go:102	Setting up own telemetry...
2024-05-13T11:23:49.018+0200	info	service@v0.100.0/telemetry.go:103	Serving metrics	{"address": ":8889", "level": "Normal"}
2024-05-13T11:23:49.018+0200	debug	exporter@v0.100.0/exporter.go:273	Alpha component. May change in the future.	{"kind": "exporter", "data_type": "metrics", "name": "file"}
2024-05-13T11:23:49.018+0200	info	exporter@v0.100.0/exporter.go:275	Development component. May change in the future.	{"kind": "exporter", "data_type": "metrics", "name": "debug"}
2024-05-13T11:23:49.018+0200	debug	receiver@v0.100.0/receiver.go:308	Beta component. May change in the future.	{"kind": "receiver", "name": "statsd", "data_type": "metrics"}
2024-05-13T11:23:49.019+0200	info	service@v0.100.0/service.go:169	Starting oteltestbedcol...	{"Version": "0.100.0-dev", "NumCPU": 12}
2024-05-13T11:23:49.019+0200	info	extensions/extensions.go:34	Starting extensions...
2024-05-13T11:23:49.020+0200	info	service@v0.100.0/service.go:195	Everything is ready. Begin running and processing data.
2024-05-13T11:23:49.020+0200	warn	localhostgate/featuregate.go:63	The default endpoints for all servers in components will change to use localhost instead of 0.0.0.0 in a future version. Use the feature gate to preview the new default.	{"feature gate ID": "component.UseLocalHostAsDefaultHost"}
2024-05-13T11:24:49.021+0200	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 1, "data points": 1}
2024-05-13T11:24:49.021+0200	info	ResourceMetrics #0
Resource SchemaURL: 
Resource attributes:
     -> source.address: Str(127.0.0.1)
     -> source.port: Str(39085)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope otelcol/statsdreceiver 0.100.0-dev
Metric #0
Descriptor:
     -> Name: test.metric
     -> Description: 
     -> Unit: 
     -> DataType: ExponentialHistogram
     -> AggregationTemporality: Delta
ExponentialHistogramDataPoints #0
Data point attributes:
     -> myKey: Str(foo)
StartTimestamp: 2024-05-13 09:24:19.020417893 +0000 UTC
Timestamp: 2024-05-13 09:24:49.021130768 +0000 UTC
Count: 1
Sum: 1.000000
Min: 1.000000
Max: 1.000000
Bucket (0.999999, 1.000000], Count: 1
	{"kind": "exporter", "data_type": "metrics", "name": "debug"}
2024-05-13T11:24:49.022+0200	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 1, "data points": 1}
2024-05-13T11:24:49.022+0200	info	ResourceMetrics #0
Resource SchemaURL: 
Resource attributes:
     -> source.address: Str(127.0.0.1)
     -> source.port: Str(40992)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope otelcol/statsdreceiver 0.100.0-dev
Metric #0
Descriptor:
     -> Name: test.metric
     -> Description: 
     -> Unit: 
     -> DataType: ExponentialHistogram
     -> AggregationTemporality: Delta
ExponentialHistogramDataPoints #0
Data point attributes:
     -> myKey: Str(foo)
StartTimestamp: 2024-05-13 09:24:19.020417893 +0000 UTC
Timestamp: 2024-05-13 09:24:49.021130768 +0000 UTC
Count: 1
Sum: 2.000000
Min: 2.000000
Max: 2.000000
Bucket (1.999999, 2.000000], Count: 1
	{"kind": "exporter", "data_type": "metrics", "name": "debug"}
2024-05-13T11:24:49.022+0200	info	ResourceMetrics #0
Resource SchemaURL: 
Resource attributes:
     -> source.address: Str(127.0.0.1)
     -> source.port: Str(36759)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope otelcol/statsdreceiver 0.100.0-dev
Metric #0
Descriptor:
     -> Name: test.metric
     -> Description: 
     -> Unit: 
     -> DataType: ExponentialHistogram
     -> AggregationTemporality: Delta
ExponentialHistogramDataPoints #0
Data point attributes:
     -> myKey: Str(bar)
StartTimestamp: 2024-05-13 09:24:19.020417893 +0000 UTC
Timestamp: 2024-05-13 09:24:49.021130768 +0000 UTC
Count: 2
Sum: 3.000000
Min: 1.000000
Max: 2.000000
Bucket (0.917004, 1.000000], Count: 1
Bucket (1.000000, 1.090508], Count: 0
Bucket (1.090508, 1.189207], Count: 0
Bucket (1.189207, 1.296840], Count: 0
Bucket (1.296840, 1.414214], Count: 0
Bucket (1.414214, 1.542211], Count: 0
Bucket (1.542211, 1.681793], Count: 0
Bucket (1.681793, 1.834008], Count: 0
Bucket (1.834008, 2.000000], Count: 1
	{"kind": "exporter", "data_type": "metrics", "name": "debug"}

And the result when using aggregate_by_source_address: false

Detaching and terminating target process
dlv dap (910618) exited with code: 0
Starting: /home/user/go/bin/dlv dap --listen=127.0.0.1:36633 --log-dest=3 from /home/user/sources/public/opentelemetry-collector-contrib/cmd/oteltestbedcol
DAP server listening at: 127.0.0.1:36633
Type 'dlv help' for list of commands.
2024-05-13T11:27:12.119+0200	info	service@v0.100.0/service.go:102	Setting up own telemetry...
2024-05-13T11:27:12.119+0200	info	service@v0.100.0/telemetry.go:103	Serving metrics	{"address": ":8889", "level": "Normal"}
2024-05-13T11:27:12.119+0200	info	exporter@v0.100.0/exporter.go:275	Development component. May change in the future.	{"kind": "exporter", "data_type": "metrics", "name": "debug"}
2024-05-13T11:27:12.119+0200	debug	exporter@v0.100.0/exporter.go:273	Alpha component. May change in the future.	{"kind": "exporter", "data_type": "metrics", "name": "file"}
2024-05-13T11:27:12.119+0200	debug	receiver@v0.100.0/receiver.go:308	Beta component. May change in the future.	{"kind": "receiver", "name": "statsd", "data_type": "metrics"}
2024-05-13T11:27:12.120+0200	info	service@v0.100.0/service.go:169	Starting oteltestbedcol...	{"Version": "0.100.0-dev", "NumCPU": 12}
2024-05-13T11:27:12.120+0200	info	extensions/extensions.go:34	Starting extensions...
2024-05-13T11:27:12.120+0200	info	service@v0.100.0/service.go:195	Everything is ready. Begin running and processing data.
2024-05-13T11:27:12.120+0200	warn	localhostgate/featuregate.go:63	The default endpoints for all servers in components will change to use localhost instead of 0.0.0.0 in a future version. Use the feature gate to preview the new default.	{"feature gate ID": "component.UseLocalHostAsDefaultHost"}
2024-05-13T11:27:42.121+0200	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 2, "data points": 2}
2024-05-13T11:27:42.122+0200	info	ResourceMetrics #0
Resource SchemaURL: 
Resource attributes:
     -> source.address: Str(0.0.0.0)
     -> source.port: Str(8125)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope otelcol/statsdreceiver 0.100.0-dev
Metric #0
Descriptor:
     -> Name: test.metric
     -> Description: 
     -> Unit: 
     -> DataType: ExponentialHistogram
     -> AggregationTemporality: Delta
ExponentialHistogramDataPoints #0
Data point attributes:
     -> myKey: Str(foo)
StartTimestamp: 2024-05-13 09:27:12.120608796 +0000 UTC
Timestamp: 2024-05-13 09:27:42.120704563 +0000 UTC
Count: 2
Sum: 3.000000
Min: 1.000000
Max: 2.000000
Bucket (0.917004, 1.000000], Count: 1
Bucket (1.000000, 1.090508], Count: 0
Bucket (1.090508, 1.189207], Count: 0
Bucket (1.189207, 1.296840], Count: 0
Bucket (1.296840, 1.414214], Count: 0
Bucket (1.414214, 1.542211], Count: 0
Bucket (1.542211, 1.681793], Count: 0
Bucket (1.681793, 1.834008], Count: 0
Bucket (1.834008, 2.000000], Count: 1
ScopeMetrics #1
ScopeMetrics SchemaURL: 
InstrumentationScope otelcol/statsdreceiver 0.100.0-dev
Metric #0
Descriptor:
     -> Name: test.metric
     -> Description: 
     -> Unit: 
     -> DataType: ExponentialHistogram
     -> AggregationTemporality: Delta
ExponentialHistogramDataPoints #0
Data point attributes:
     -> myKey: Str(bar)
StartTimestamp: 2024-05-13 09:27:12.120608796 +0000 UTC
Timestamp: 2024-05-13 09:27:42.120704563 +0000 UTC
Count: 2
Sum: 3.000000
Min: 1.000000
Max: 2.000000
Bucket (0.917004, 1.000000], Count: 1
Bucket (1.000000, 1.090508], Count: 0
Bucket (1.090508, 1.189207], Count: 0
Bucket (1.189207, 1.296840], Count: 0
Bucket (1.296840, 1.414214], Count: 0
Bucket (1.414214, 1.542211], Count: 0
Bucket (1.542211, 1.681793], Count: 0
Bucket (1.681793, 1.834008], Count: 0
Bucket (1.834008, 2.000000], Count: 1
	{"kind": "exporter", "data_type": "metrics", "name": "debug"}

Documentation:

  • Added the new resource attributes to the metadata file
  • Describe the new config option and recommendation when disabling the default

@Manuelraa Manuelraa requested review from jmacd and dmitryax as code owners May 13, 2024 09:31
@Manuelraa Manuelraa requested a review from a team May 13, 2024 09:31
@github-actions github-actions bot added the receiver/statsd statsd related issues label May 13, 2024
@Manuelraa Manuelraa mentioned this pull request May 13, 2024
Closed
@github-actions GitHub Actions
Copy link
Contributor

This PR was marked stale due to lack of activity. It will be closed in 14 days.

@github-actions github-actions bot added the Stale label May 28, 2024
@github-actions GitHub Actions
Copy link
Contributor

Closed as inactive. Feel free to reopen if this PR is still being worked on.

@github-actions github-actions bot closed this Jun 11, 2024
@alain-owen-kuchta
Copy link

Hey! Just want to bump that this fixes a big headache for my project. Would love to see this merged!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jmacd jmacd Awaiting requested review from jmacd

@dmitryax dmitryax Awaiting requested review from dmitryax dmitryax is a code owner

Assignees

@evan-bradley evan-bradley

Labels
receiver/statsd statsd related issues Stale
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[statsdreceiver] incorrect aggregation when several producers report the same metrics
3 participants
@Manuelraa @alain-owen-kuchta @evan-bradley