-
Notifications
You must be signed in to change notification settings - Fork 524
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
docs: update Logstash output docs #11165
Conversation
This pull request does not have a backport label. Could you fix it @bmorelli25? 🙏
NOTE: |
In addition to @metadata, APM Server provides other potentially useful fields, like the | ||
`processor.event` field and `data_stream` fields, which can be used to conditionally operate on | ||
{apm-guide-ref}/data-model.html[event types], namespaces, or datasets. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewers: I removed the custom index example because it was outdated. It would be great to come up with a new example to provide here. What is a common use-case you see with Logstash?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe using logstash for routing events to different data streams can still be a fair use case; e.g. routing all metrics
events to the same metrics datastream, rather than to service specific ones for custom metrics.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! I was able to get this working with the following config. I've updated the documentation to match.
output {
if [@metadata][beat] == "apm-server" {
if [processor][event] == "metric" {
elasticsearch {
index => "%{[data_stream][type]}-custom-%{[data_stream][namespace]}"
action => "create"
cloud_id => "${CLOUD_ID}"
cloud_auth => "${CLOUD_AUTH}"
}
} else {
elasticsearch {
data_stream => "true"
cloud_id => "${CLOUD_ID}"
cloud_auth => "${CLOUD_AUTH}"
}
}
}
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good, thanks!
Waiting with approval until the bit for the @metadata
or processor.event
use case is finalised.
In addition to @metadata, APM Server provides other potentially useful fields, like the | ||
`processor.event` field and `data_stream` fields, which can be used to conditionally operate on | ||
{apm-guide-ref}/data-model.html[event types], namespaces, or datasets. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe using logstash for routing events to different data streams can still be a fair use case; e.g. routing all metrics
events to the same metrics datastream, rather than to service specific ones for custom metrics.
* docs: update ls docs * Add custom metric example --------- Co-authored-by: Silvia Mitter <silvia.mitter@elastic.co> (cherry picked from commit 448874c)
* docs: update ls docs * Add custom metric example --------- Co-authored-by: Silvia Mitter <silvia.mitter@elastic.co> (cherry picked from commit 448874c)
* docs: update ls docs * Add custom metric example --------- Co-authored-by: Silvia Mitter <silvia.mitter@elastic.co>
Summary
@metadata.beat
toapm-server
@metadata.pipeline
. This was previously documented asapm
, however, when I set up Logstash and usedstdout { codec => rubydebug { metadata => true }
to output@metadata
, I only saw the following:input
recommendation from the Beats input plugin to the Elastic Agent input plugindata_stream.*
fieldsDocumentation preview
View a preview of these changes by clicking here.
Resources