Added adhoc queries and snippet

.github/workflows/adhoc-remote-benchmarks.yml

@@ -22,7 +22,7 @@ on:
      test_package:
        description: 'Benchmark Test Package'
        required: true
-       default: 'io.deephaven.benchmark.tests'
+       default: 'io.deephaven.benchmark.tests.standard'
        type: string
      test_class_regex:
        description: 'Benchmark Test Class Pattern'
@@ -47,3 +47,4 @@ jobs:
       test_class_regex: ${{ inputs.test_class_regex }}
       scale_row_count: ${{ inputs.scale_row_count }} 
     secrets: inherit
+

docs/PublishedResults.md

@@ -4,6 +4,7 @@ Many Deephaven benchmarks are published to the [deephaven-benchmark](https://sto
 accessible only through Google's storage API. Benchmarks are organized in the bucket in much the same ways as the 
 [results generated from the command line](CollectedResults.md).
 
+## Main Query Snippet
 The easiest way to access the published benchmark results is by running the following Deephaven query code snippet
 in an instance of the Deephaven Engine.
 
@@ -19,18 +20,42 @@ with urlopen(root + '/deephaven-benchmark/benchmark_tables.dh.py') as r:
 ````
 
 This will process the available benchmarks for the given benchmark category (release or nightly), merge test runs together, and generate some 
-useful Deephaven tables that can be used to explore the benchmarks.
+useful Deephaven tables that can be used to explore the benchmarks. This is the main query snippet that other query snippets below rely on.
 
 Requirements:
-- [Deephaven 0.23.0 or higher](https://deephaven.io/core/docs/tutorials/quickstart/)
+- [Deephaven 0.32.0 or higher](https://deephaven.io/core/docs/tutorials/quickstart/)
 - 4G RAM allocated to Deephaven Server VM (The Default)
 
-## Tables Generated By The Script
-
+Tables Generated by the Main Query Snippet
 - bench_results: A merge of all available benchmark runs for a category
 - bench_platforms: A merge the JVM configuration and hardware for the benchmark runs
 - bench_metrics: A merge of the JVM metrics taken before and after each running
 - bench_metrics_diff: The difference of the before and after metrics for each benchmark
 - bench_results_diff: Bench results with some metrics diffs added as columns
 - bench_results_change: Bench results with analysis of variability and rate change compared to past runs
 
+## Adhoc Query Snippet
+The adhoc query snippet is for use by developers who have executed on-demand benchmarks runs. These usually appear during an investigation
+of a subset of benchmarks. Only Deephaven developers can make these runs, but they are still publicly available in the
+[deephaven-benchmark/adhoc](https://storage.googleapis.com/deephaven-benchmark) GCloud bucket.
+(Note: These benchmark sets are often deleted after an investigation is over.)
+
+````
+from urllib.request import urlopen; import os
+
+root = 'file:///nfs' if os.path.exists('/nfs/deephaven-benchmark') else 'https://storage.googleapis.com'
+with urlopen(root + '/deephaven-benchmark/adhoc_tables.dh.py') as r:
+    benchmark_sets_arg = ['user1/setname1','user1/setname2']  # The set names to run including user (ex. ['user1/myset1','user1/myset2'])
+    benchmark_set_runs_arg = 5  # Maximum number of runs to include per set
+    exec(r.read().decode(), globals(), locals())
+````
+This will process two or more adhoc benchmark sets and produce a table the compares the rates in each. Loading an caching of the benchmark
+runs is done automatically.
+
+Requirements:
+- [Deephaven 0.32.0 or higher](https://deephaven.io/core/docs/tutorials/quickstart/)
+- 4G RAM allocated to Deephaven Server VM (The Default)
+
+Tables Generated by the Adhoc Query Snippet
+- adhoc_set_compare: Shows benchmarks, rate and variability columns for each benchmark set
+

src/main/resources/io/deephaven/benchmark/run/profile/queries/adhoc_tables.dh.py

@@ -0,0 +1,58 @@
+# Copyright (c) 2022-2024 Deephaven Data Labs and Patent Pending 
+#
+# Supporting Deephaven queries to use the benchmark_snippet to investigate changes between two or more
+# adhoc benchmark set runs
+# - Make a table containing rates and diffs for the given benchmark sets
+# - Expects the following arguments set in globals() before execution
+#   - benchmark_sets_arg = []  # The set names to run including user (ex. ['user1/myset1','user1/myset2')
+#   - benchmark_set_runs_arg = 5  # Number of runs to load from each set (Can be greater than available)
+# Requirements: Deephaven 0.32.0 or greater
+
+from urllib.request import urlopen; import os, re
+
+benchmark_sets_prefix =  os.path.commonprefix(benchmark_sets_arg)
+
+def normalize_name(name):
+    name = name.replace('/','__')
+    return re.sub('[^A-Za-z0-9_$]', '_', name)
+
+result = None
+first_set = None
+for benchmark_set in benchmark_sets_arg:
+    root = 'file:///nfs' if os.path.exists('/nfs/deephaven-benchmark') else 'https://storage.googleapis.com'
+    with urlopen(root + '/deephaven-benchmark/benchmark_tables.dh.py') as r:
+        benchmark_storage_uri_arg = root + '/deephaven-benchmark'
+        benchmark_category_arg ='adhoc/' + benchmark_set
+        benchmark_max_runs_arg = benchmark_set_runs_arg
+        exec(r.read().decode(), globals(), locals())
+
+    set_name = normalize_name(benchmark_set.replace(benchmark_sets_prefix,''))
+    tbl = bench_results_change.group_by(['benchmark_name']).view([
+        'Benchmark=benchmark_name',
+        'Variability__' + set_name + '=(float)rstd(op_rate) / 100.0',
+        'Rate__' + set_name + '=(long)median(op_rate)'
+    ])
+    if result is None:
+        result = tbl
+        first_set = set_name
+    else:
+        first_rate = 'Rate__' + first_set
+        curr_rate = 'Rate__' + set_name
+        result = result.join(tbl, on=['Benchmark'], joins=['Variability__' + set_name, curr_rate])
+        result = result.update_view([
+            'Change__' + set_name + '=(float)gain(' + first_rate + ',' + curr_rate + ') / 100.0'
+        ])
+
+bench_results = bench_metrics = bench_platforms = bench_metrics_diff = None
+bench_results_diff = bench_results_change = tbl = None
+
+column_formats = []
+for col in result.columns:
+    n = col.name
+    if n.startswith('Variability') or  n.startswith('Change'):
+        column_formats.append(n + '=Decimal(`0.0%`)')
+    if n.startswith('Rate'):
+        column_formats.append(n + '=Decimal(`###,##0`)')
+
+adhoc_set_compare = result.format_columns(column_formats)
+result = None