docs: update Logstash output docs

[bmorelli25]

Summary

• Updates the default value of @metadata.beat to apm-server
• Removes the default value of @metadata.pipeline. This was previously documented as apm, however, when I set up Logstash and used stdout { codec => rubydebug { metadata => true } to output @metadata, I only saw the following:

  "@metadata"=>{
    "version"=>"8.8.2",
    "input"=>{
      "beats"=>{
        "host"=>{
          "ip"=>"0:0:0:0:0:0:0:1"
        }
      }
    },
    "beat"=>"apm-server",
    "type"=>"_doc"
  }

• Changed the default input recommendation from the Beats input plugin to the Elastic Agent input plugin
• Documented how (and why) to dedot data_stream.* fields
• Documented how to enable data stream routing

Documentation preview

View a preview of these changes by clicking here.

Resources

• Closes Update documentation for logstash output with the correct defaults #10643
• https://github.com/elastic/sdh-apm/issues/983

[mergify]

This pull request does not have a backport label. Could you fix it @bmorelli25? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-7.x is the label to automatically backport to the 7.x branch.
• backport-7./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[apmmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-07-24T16:01:49.587+0000

• Duration: 6 min 4 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate and publish the docker images.

• /test windows : Build & tests on Windows.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[bmorelli25]

Reviewers: I removed the custom index example because it was outdated. It would be great to come up with a new example to provide here. What is a common use-case you see with Logstash?

[simitt]

I believe using logstash for routing events to different data streams can still be a fair use case; e.g. routing all metrics events to the same metrics datastream, rather than to service specific ones for custom metrics.

[bmorelli25]

Thanks! I was able to get this working with the following config. I've updated the documentation to match.

output {
  if [@metadata][beat] == "apm-server" {
    if [processor][event] == "metric" {
      elasticsearch {
        index => "%{[data_stream][type]}-custom-%{[data_stream][namespace]}"
        action => "create"
        cloud_id => "${CLOUD_ID}"
        cloud_auth => "${CLOUD_AUTH}"
      }
    } else {
      elasticsearch {
        data_stream => "true"
        cloud_id => "${CLOUD_ID}"
        cloud_auth => "${CLOUD_AUTH}"
      }
    }
  }
}

[simitt]

Looks good, thanks!
Waiting with approval until the bit for the @metadata or processor.event use case is finalised.

[simitt]

I believe using logstash for routing events to different data streams can still be a fair use case; e.g. routing all metrics events to the same metrics datastream, rather than to service specific ones for custom metrics.