-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Confirm span merge span name drop strategy #54
Comments
@axw is this something that needs to be fixed before releasing LSM based aggregation in apm-server? |
The behaviour has changed, so yes I think we should make a decision about whether to keep it before releasing 8.10. (I think the new behaviour is probably fine though.) |
Old behavior in apm-server 8.9 (code)
Existing apm-aggregation behavior (code)
Observations:
Question:
|
The span name drop strategy (which I regret, should have introduced a new metric) is meant to protect against high-cardinality span names produced by a service, not protecting against cardinality across all the services. I think the change is fine, and we're still protecting against that. If we were to take into account the global limit, then services producing high cardinality span names would penalise other services, by causing them to drop the span name. I don't think that is desirable.
This seems off. Where is that code? I think we should probably be counting with the span name, so we can see which services are producing high cardinality span names. |
It is this line here, which happens only after we modify fromSpan.Key here. It is nothing new, since this apparently comes from the similar looking code in 8.9 apm-server aggregation.
I agree, that's why I raised the question in the first place.
I believe we want to change the last point to always estimate cardinality with full unmodified span key. |
@carsonip 👍 agreed. @lahsivjar any concerns with that? |
No concerns. SGTM! |
With existing code in https://github.com/elastic/apm-aggregation/blob/main/aggregators/merger.go#L263 we are dropping span name even if global limit is reached first. Check if this is desirable.
The text was updated successfully, but these errors were encountered: