bench: Add ndjson benchmarks

[carsonip]

Add benchmarks that take ndjson in /aggregators/testdata/ and run aggregator on it. No ndjson file is added to the PR. Users should add ndjson file manually to a gitignored directory /aggregators/testdata/. ndjson files can be downloaded from apm-perf repo.

[endorama]

Benchmarks look good to me!

The only change I can't confirm, as it's not clear to me the way it affects other tests or the benchmark themselves, is changing the newTestAggregator to use in memory instead of a data dir.
My understanding is that we use the filesystem in production and is not clear to me if this change is expected, so I'd like another pair of eyes there.

[endorama]

@carsonip as you mention ndjson test files are not part of this PR would be nice a link to get them or pointing to some documentation. I got them from this PR but that is not practical in the short term.

[carsonip]

Thanks for the review.

The only change I can't confirm, as it's not clear to me the way it affects other tests or the benchmark themselves, is changing the newTestAggregator to use in memory instead of a data dir.

I originally planned to make benchmarks more stable by using the in-memory option. However, that's not within the scope of this PR and I think it is reasonable to do it in another PR. We'll eventually have to benchmark in-memory performance as apm-server should use the option.

as you mention ndjson test files are not part of this PR would be nice a link to get them or pointing to some documentation

Good call. Done.

[lahsivjar]

I originally planned to make benchmarks more stable by using the in-memory option. However, that's not within the scope of this PR and I think it is reasonable to do it in another PR. We'll eventually have to benchmark in-memory performance as apm-server should use the option.

We can probably write a simple wrapper and make all benchmarks run for both in-memory and disk.