Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DEBUG-2647 Run multiple Ruby micro-benchmark files #3810

Merged
merged 10 commits into from
Aug 5, 2024
Merged

DEBUG-2647 Run multiple Ruby micro-benchmark files #3810

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
8bec446
separate gem loading benchmark
p Jul 25, 2024
80def75
run all benchmarks
p Jul 25, 2024
2840239
use same pattern for profiler benchmark save paths (benchmarks subdir)
p Jul 25, 2024
d401d5b
explicitly list all benchmarks
p Jul 31, 2024
9015359
validate all of the benchmark files for lack of errors
p Jul 31, 2024
af9da85
add readme with instructions how to add a new benchmark file
p Jul 31, 2024
9d052cb
move benchmark validation spec to top level because it is no longer t…
p Jul 31, 2024
4e83fae
Merge branch 'master' into benchmark-gem-loading
p Aug 5, 2024
42b07cd
reorganize benchmark validators again
p Aug 5, 2024
ede6784
rubocop
p Aug 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions benchmarks/gem_loading.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Used to quickly run benchmark under RSpec as part of the usual test suite, to validate it didn't bitrot
VALIDATE_BENCHMARK_MODE = ENV['VALIDATE_BENCHMARK'] == 'true'

return unless __FILE__ == $PROGRAM_NAME || VALIDATE_BENCHMARK_MODE

require 'open3'

class GemLoadingBenchmark
def benchmark_gem_loading
# This benchmark needs to be run in a clean environment where datadog is
# not loaded yet.
#
# Now that this benchmark is in its own file, it does not need
# to spawn a subprocess IF we would always execute this benchmark
# file by itself.
output, status = Open3.capture2e('bundle', 'exec', 'ruby', stdin_data: <<-RUBY)
raise "Datadog is already loaded" if defined?(::Datadog::Core)

lib = File.expand_path('../lib', '#{__dir__}')
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)

VALIDATE_BENCHMARK_MODE = #{VALIDATE_BENCHMARK_MODE}
require 'benchmark/ips'

Benchmark.ips do |x|
# Gem loading is quite slower than the other microbenchmarks
benchmark_time = VALIDATE_BENCHMARK_MODE ? { time: 0.001, warmup: 0 } : { time: 60, warmup: 5 }
x.config(**benchmark_time)

x.report("Gem loading") do
pid = fork { require 'datadog' }

_, status = Process.wait2(pid)
raise unless status.success?
end

x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end
RUBY

print output

raise "Benchmark failed with status #{status}: #{output}" unless status.success?
end
end

puts "Current pid is #{Process.pid}"

GemLoadingBenchmark.new.instance_exec do
benchmark_gem_loading
end
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't forget to update https://github.com/DataDog/dd-trace-rb/blob/master/spec/datadog/tracing/validate_benchmarks_spec.rb#L12 :)

(Technically this is not a tracing-specific thing but I think it's fine to keep it there for now? -- as long as we test it somewhere)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a readme to the benchmarks directory with this point as a reminder.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And moved the file to top level since it validates all benchmarks, not just tracing.

4 changes: 2 additions & 2 deletions benchmarks/profiler_allocation.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ def run_benchmark

x.report('Allocations (baseline)', 'BasicObject.new')

x.save! 'profiler-allocation-results.json' unless VALIDATE_BENCHMARK_MODE
x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end

Expand All @@ -53,7 +53,7 @@ def run_benchmark

x.report("Allocations (#{ENV['CONFIG']})", 'BasicObject.new')

x.save! 'profiler-allocation-results.json' unless VALIDATE_BENCHMARK_MODE
x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end
end
Expand Down
12 changes: 6 additions & 6 deletions benchmarks/profiler_gc.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ def run_benchmark
Datadog::Profiling::Collectors::ThreadContext::Testing._native_sample_after_gc(@collector)
end

x.save! 'profiler-gc-results.json' unless VALIDATE_BENCHMARK_MODE
x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end

Expand All @@ -72,7 +72,7 @@ def run_benchmark
@recorder.serialize
end

x.save! 'profiler-gc-minute-results.json' unless VALIDATE_BENCHMARK_MODE
x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end

Expand All @@ -85,7 +85,7 @@ def run_benchmark

x.report('Major GC runs (profiling disabled)', 'GC.start')

x.save! 'profiler-gc-integration-results.json' unless VALIDATE_BENCHMARK_MODE
x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end

Expand All @@ -105,7 +105,7 @@ def run_benchmark

x.report('Major GC runs (profiling enabled)', 'GC.start')

x.save! 'profiler-gc-integration-results.json' unless VALIDATE_BENCHMARK_MODE
x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end

Expand All @@ -120,7 +120,7 @@ def run_benchmark

x.report('Allocations (profiling disabled)', 'Object.new')

x.save! 'profiler-gc-integration-allocations-results.json' unless VALIDATE_BENCHMARK_MODE
x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end

Expand All @@ -140,7 +140,7 @@ def run_benchmark

x.report('Allocations (profiling enabled)', 'Object.new')

x.save! 'profiler-gc-integration-allocations-results.json' unless VALIDATE_BENCHMARK_MODE
x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/profiler_hold_resume_interruptions.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ def run_benchmark
Datadog::Profiling::Collectors::CpuAndWallTimeWorker._native_resume_signals
end

x.save! 'profiler_hold_resume_interruptions-results.json' unless VALIDATE_BENCHMARK_MODE
x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end
end
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/profiler_http_transport.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ def run_benchmark
run_once
end

x.save! 'profiler-http-transport-results.json' unless VALIDATE_BENCHMARK_MODE
x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end
end
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/profiler_memory_sample_serialize.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ def run_benchmark
recorder.serialize
end

x.save! 'profiler_memory_sample_serialize-results.json' unless VALIDATE_BENCHMARK_MODE
x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end
end
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/profiler_sample_loop_v2.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ def run_benchmark
Datadog::Profiling::Collectors::ThreadContext::Testing._native_sample(@collector, PROFILER_OVERHEAD_STACK_THREAD)
end

x.save! 'profiler-sample-loop-v2-results.json' unless VALIDATE_BENCHMARK_MODE
x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/profiler_sample_serialize.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ def run_benchmark
nil
end

x.save! 'profiler_sample_serialize-results.json' unless VALIDATE_BENCHMARK_MODE
x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end

Expand Down
14 changes: 14 additions & 0 deletions benchmarks/run_all.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#!/bin/sh

# This script is invoked by benchmarking-platform shell scripts
# to run all of the benchmarks defined in the tracer.

set -ex

for file in \
`dirname "$0"`/tracing_trace.rb \
`dirname "$0"`/gem_loading.rb \
`dirname "$0"`/profiler_*.rb;
do
bundle exec ruby "$file"
done
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I don't think it's blocking for this PR, but I wonder if we should try to put in some kind of check for files that aren't run.

E.g. something along the lines of this check?

Otherwise, it can be a bit confusing that if anyone adds a new profiler benchmark, it gets automatically picked up, but if you add any other kind of benchmark, it doesn't?

Alternatively, should we just run every .rb file on the benchmark/ folder, and move anything that's not a benchmark to a lib/ subdirectory or something like that? (Or maybe list here the files that shouldn't be run instead of the ones that should?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was actually not sure what the right approach to invoking "all" benchmarks was. Let me enumerate the files explicitly in this PR and try to improve the situation later.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move of supporting files to lib is actually part of #3787 which will be on top of this PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds reasonable -- now there's less magic, but it's a bit more consistent 👍

35 changes: 0 additions & 35 deletions benchmarks/tracing_trace.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@
return unless __FILE__ == $PROGRAM_NAME || VALIDATE_BENCHMARK_MODE

require 'benchmark/ips'
require 'open3'
require 'datadog'

class TracingTraceBenchmark
Expand Down Expand Up @@ -177,39 +176,6 @@ def benchmark_propagation_trace_context
end
end
end

def benchmark_gem_loading
# This benchmark needs to be run in a clean environment where datadog is not loaded yet
output, status = Open3.capture2e('bundle', 'exec', 'ruby', stdin_data: <<-RUBY)
raise "Datadog is already loaded" if defined?(::Datadog::Core)

lib = File.expand_path('../lib', '#{__dir__}')
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)

VALIDATE_BENCHMARK_MODE = #{VALIDATE_BENCHMARK_MODE}
require 'benchmark/ips'

Benchmark.ips do |x|
# Gem loading is quite slower than the other microbenchmarks
benchmark_time = VALIDATE_BENCHMARK_MODE ? { time: 0.001, warmup: 0 } : { time: 60, warmup: 5 }
x.config(**benchmark_time)

x.report("Gem loading") do
pid = fork { require 'datadog' }

_, status = Process.wait2(pid)
raise unless status.success?
end

x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end
RUBY

print output

raise "Benchmark failed with status #{status}: #{output}" unless status.success?
end
end

puts "Current pid is #{Process.pid}"
Expand All @@ -230,5 +196,4 @@ def run_benchmark(&block)
run_benchmark { benchmark_to_digest_continue }
run_benchmark { benchmark_propagation_datadog }
run_benchmark { benchmark_propagation_trace_context }
run_benchmark { benchmark_gem_loading }
end
Loading