Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[exporter/elasticsearch] Add sanitization utils for datastream fields #35494

Merged

Conversation

rubvs
Copy link
Contributor

@rubvs rubvs commented Sep 30, 2024

Description: Adds util functions to sanitize data stream fields

Link to tracking Issue: Fixes #34285

Testing: Added unit tests

Copy link

linux-foundation-easycla bot commented Sep 30, 2024

CLA Signed

The committers listed above are authorized under a signed CLA.

Copy link
Contributor

@carsonip carsonip left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, looks good functionality wise. A couple of minor comments.

exporter/elasticsearchexporter/data_stream_router.go Outdated Show resolved Hide resolved
exporter/elasticsearchexporter/data_stream_router.go Outdated Show resolved Hide resolved
exporter/elasticsearchexporter/util.go Outdated Show resolved Hide resolved
exporter/elasticsearchexporter/exporter_test.go Outdated Show resolved Hide resolved
exporter/elasticsearchexporter/exporter_test.go Outdated Show resolved Hide resolved
exporter/elasticsearchexporter/exporter_test.go Outdated Show resolved Hide resolved
exporter/elasticsearchexporter/util.go Outdated Show resolved Hide resolved
exporter/elasticsearchexporter/exporter_test.go Outdated Show resolved Hide resolved
@rubvs
Copy link
Contributor Author

rubvs commented Oct 1, 2024

@carsonip I have improved the sanitizeDataStreamField to leverage inlining. I also merged the .otel suffix extension into the sanitization check, since I have to take the extension into account when doing the 100 char check. Let me know if you approve this approach.

@rubvs rubvs requested a review from carsonip October 1, 2024 14:27
@rubvs
Copy link
Contributor Author

rubvs commented Oct 1, 2024

@carsonip I also added a test to model_test.go. I do not know if this is needed, but I think it won't hurt to add it either. If you want me to keep this, I'll update the test to get rid of the disallowedRunes and replace if with the hardcoded runes, as you suggested in the previous comments.

@rubvs rubvs changed the title Add sanitization utils for datastream fields [exporter/elasticsearch] Add sanitization utils for datastream fields Oct 1, 2024
Copy link
Contributor

@carsonip carsonip left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice catch about the otel suffix! a comment about refactoring

exporter/elasticsearchexporter/data_stream_router.go Outdated Show resolved Hide resolved
exporter/elasticsearchexporter/data_stream_router.go Outdated Show resolved Hide resolved
exporter/elasticsearchexporter/data_stream_router.go Outdated Show resolved Hide resolved
exporter/elasticsearchexporter/data_stream_router.go Outdated Show resolved Hide resolved
@rubvs rubvs marked this pull request as ready for review October 1, 2024 16:40
@rubvs rubvs requested a review from a team as a code owner October 1, 2024 16:40
@rubvs rubvs requested a review from carsonip October 2, 2024 13:21
@rubvs rubvs requested a review from crobert-1 October 3, 2024 01:18
@crobert-1 crobert-1 added the ready to merge Code review completed; ready to merge by maintainers label Oct 3, 2024
Copy link
Member

@andrzej-stencel andrzej-stencel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be documented?

@rubvs
Copy link
Contributor Author

rubvs commented Oct 8, 2024

Shouldn't this be documented?

Hi @andrzej-stencel, I've updated the readme of ES exported to reference the restriction docs. Let me know if this suffices, or if you think I should articulate things differently.

@andrzej-stencel
Copy link
Member

Shouldn't this be documented?

I've updated the readme of ES exporter to reference the restriction docs. Let me know if this suffices, or if you think I should articulate things differently.

Thanks @rubvs, this is better than nothing. Ideally the docs would mention that the restricted characters will be replaced with underscores _.

@atoulme
Copy link
Contributor

atoulme commented Oct 11, 2024

Please resolve conflicts.

@atoulme atoulme removed the ready to merge Code review completed; ready to merge by maintainers label Oct 11, 2024
@andrzej-stencel andrzej-stencel merged commit a39906a into open-telemetry:main Oct 15, 2024
158 checks passed
@github-actions github-actions bot added this to the next release milestone Oct 15, 2024
@rubvs rubvs deleted the elastic-sanitize-datastream-fields branch October 15, 2024 16:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[exporter/elasticsearch] Data stream routing does not sanitize values
7 participants