[NO-TICKET] Raise benchmark default durations

[ivoanjo]

What does this PR do?

This PR goes through our existing benchmarks, and for those that used a default duration of 10 or 12 seconds, raises the duration to 30s.

Motivation:

We've observed most of these benchmarks having flaky regressions/improvements on PRs that shouldn't affect them.

I suspect that the running the benchmarks for such short time may be contributing to flakiness. For instance, one of the benchmarks that flakes the most often is profiler_sample_serialize.rb, and concidentally that's a benchmark where each iteration does a lot of work, and thus in a typical 10 second run we may only see around 70 iterations.

Hopefully by running the benchmarks for slightly longer we'll have more consistent results.

Additional Notes:

There is a downside to this -- because our benchmarks are currently executed sequentially in CI, this will make the benchmark run took quite longer than it used to.

Hopefully this trade-off is reasonable; if not, we can re-evaluate.

How to test the change?

Check the latest benchmarks CI run, and confirm the tests are running for 30 seconds, rather than 10/12.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.85%. Comparing base (3679939) to head (ee157a1).
Report is 10 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3869      +/-   ##
==========================================
- Coverage   97.85%   97.85%   -0.01%     
==========================================
  Files        1271     1277       +6     
  Lines       76024    76304     +280     
  Branches     3740     3740              
==========================================
+ Hits        74395    74667     +272     
- Misses       1629     1637       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-08-28 17:05:59

Comparing candidate commit ee157a1 in PR branch ivoanjo/raise-benchmark-durations with baseline commit 0c87de3 in branch master.

Found 2 performance improvements and 2 performance regressions! Performance is the same for 21 metrics, 0 unstable metrics.

scenario:profiler - Major GC runs (profiling disabled)

• 🟩 throughput [+2.789op/s; +2.906op/s] or [+2.395%; +2.495%]

scenario:profiler - Major GC runs (profiling enabled)

• 🟩 throughput [+2.636op/s; +2.749op/s] or [+2.389%; +2.491%]

scenario:profiler - sample timeline=false

• 🟥 throughput [-0.452op/s; -0.439op/s] or [-6.267%; -6.084%]

scenario:tracing - Propagation - Datadog

• 🟥 throughput [-737.799op/s; -696.846op/s] or [-2.217%; -2.094%]

[p-datadog]

I support the changes if they make the benchmarks less flaky, but the runtime is now 45 minutes for this configuration? Should we consider splitting it?

[ivoanjo]

I support the changes if they make the benchmarks less flaky, but the runtime is now 45 minutes for this configuration? Should we consider splitting it?

Good point. To be honest, I'm still seeing too much variation on the benchmarks, so as a last desperate measure I'll try raising every one to 1 minute.

If one minute is reasonable (I'm starting to suspect it's not going to be), I think it's worth looking into the splitting...

[ivoanjo]

Ok, it seems like even with 60 seconds duration, we're still seeing bogus improvements/regressions being reported. Let me move this PR to draft for now while I follow-up on other solutions.

[ivoanjo]

We ended up tweaking the thresholds on the benchmarking platform and it reduced the variance a lot. There's still a few tests that seem to show wider variance -- I'll open up separate PRs for tweaking their duration as needed.

Closing this one for now.