Skip to content

Commit

Permalink
[NO-TICKET] Don't put results in benchmarking folder directly
Browse files Browse the repository at this point in the history
**What does this PR do?**

This PR tweaks the configuration for benchmark output results to place
result files in the current directory the benchmark gets executed from,
instead of placing the results in the benchmarks directory.

**Motivation:**

In #3810 we standardized the results output file name in benchmarks to
match the benchmark name.

In that PR, we used `__FILE__` as a prefix. This changed where results
are placed: where previously they were placed in the current folder
where the benchmarks were run from (often the root of the repo), with
this PR, they started getting placed in the benchmarks directory.

This clashes with our `validate_benchmarks_spec.rb` that look for
files in those directories, e.g. running a benchmark and then running
the test suite will make the test suite fail which is somewhat
annoying.

While I could've changed the tests to filter out results files, I also
find it useful to place the results where I'm executing the benchmarks
from, as it makes organization easier (you just run the benchmark from
where you want and you get the result there ).

**Additional Notes:**

N/A

**How to test the change?**

Run the benchmarks and confirm the results file gets placed in the
current folder! :)
  • Loading branch information
ivoanjo committed Aug 8, 2024
1 parent 0ba87b9 commit b971c15
Show file tree
Hide file tree
Showing 9 changed files with 21 additions and 21 deletions.
2 changes: 1 addition & 1 deletion benchmarks/library_gem_loading.rb
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ def benchmark_gem_loading
raise unless status.success?
end
x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.save! "#{File.basename(__FILE__)}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end
RUBY
Expand Down
4 changes: 2 additions & 2 deletions benchmarks/profiler_allocation.rb
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ def run_benchmark

x.report('Allocations (baseline)', 'BasicObject.new')

x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.save! "#{File.basename(__FILE__)}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end

Expand All @@ -50,7 +50,7 @@ def run_benchmark

x.report("Allocations (#{ENV['CONFIG']})", 'BasicObject.new')

x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.save! "#{File.basename(__FILE__)}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end
end
Expand Down
12 changes: 6 additions & 6 deletions benchmarks/profiler_gc.rb
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ def run_benchmark
Datadog::Profiling::Collectors::ThreadContext::Testing._native_sample_after_gc(@collector)
end

x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.save! "#{File.basename(__FILE__)}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end

Expand All @@ -69,7 +69,7 @@ def run_benchmark
@recorder.serialize
end

x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.save! "#{File.basename(__FILE__)}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end

Expand All @@ -81,7 +81,7 @@ def run_benchmark

x.report('Major GC runs (profiling disabled)', 'GC.start')

x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.save! "#{File.basename(__FILE__)}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end

Expand All @@ -100,7 +100,7 @@ def run_benchmark

x.report('Major GC runs (profiling enabled)', 'GC.start')

x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.save! "#{File.basename(__FILE__)}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end

Expand All @@ -114,7 +114,7 @@ def run_benchmark

x.report('Allocations (profiling disabled)', 'Object.new')

x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.save! "#{File.basename(__FILE__)}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end

Expand All @@ -133,7 +133,7 @@ def run_benchmark

x.report('Allocations (profiling enabled)', 'Object.new')

x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.save! "#{File.basename(__FILE__)}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/profiler_hold_resume_interruptions.rb
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ def run_benchmark
Datadog::Profiling::Collectors::CpuAndWallTimeWorker._native_resume_signals
end

x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.save! "#{File.basename(__FILE__)}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end
end
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/profiler_http_transport.rb
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ def run_benchmark
run_once
end

x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.save! "#{File.basename(__FILE__)}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end
end
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/profiler_memory_sample_serialize.rb
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ def run_benchmark
recorder.serialize
end

x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.save! "#{File.basename(__FILE__)}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end
end
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/profiler_sample_loop_v2.rb
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ def run_benchmark
Datadog::Profiling::Collectors::ThreadContext::Testing._native_sample(@collector, PROFILER_OVERHEAD_STACK_THREAD)
end

x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.save! "#{File.basename(__FILE__)}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/profiler_sample_serialize.rb
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ def run_benchmark
nil
end

x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.save! "#{File.basename(__FILE__)}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end

Expand Down
14 changes: 7 additions & 7 deletions benchmarks/tracing_trace.rb
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ def trace(x, depth)
trace(x, 10)
trace(x, 100)

x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.save! "#{File.basename(__FILE__)}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end
end
Expand Down Expand Up @@ -74,7 +74,7 @@ def trace(x, depth)
trace(x, 10)
trace(x, 100)

x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.save! "#{File.basename(__FILE__)}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end
end
Expand All @@ -88,7 +88,7 @@ def benchmark_to_digest
trace.to_digest
end

x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.save! "#{File.basename(__FILE__)}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end
end
Expand All @@ -103,7 +103,7 @@ def benchmark_log_correlation
Datadog::Tracing.log_correlation
end

x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.save! "#{File.basename(__FILE__)}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end
end
Expand All @@ -119,7 +119,7 @@ def benchmark_to_digest_continue
Datadog::Tracing.continue_trace!(digest)
end

x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.save! "#{File.basename(__FILE__)}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end
end
Expand Down Expand Up @@ -148,7 +148,7 @@ def benchmark_propagation_datadog
raise unless extracted_trace_digest
end

x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.save! "#{File.basename(__FILE__)}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end
end
Expand All @@ -171,7 +171,7 @@ def benchmark_propagation_trace_context
raise unless extracted_trace_digest
end

x.save! "#{__FILE__}-results.json" unless VALIDATE_BENCHMARK_MODE
x.save! "#{File.basename(__FILE__)}-results.json" unless VALIDATE_BENCHMARK_MODE
x.compare!
end
end
Expand Down

0 comments on commit b971c15

Please sign in to comment.