This document provides a high-level overview of how to contribute to
Renaissance Benchmark Suite.
Details are described in standalone files in the documentation
subdirectory inside project repository (tarball).
We would also like to point you to the Renaissance Code of Conduct. As a member of the Renaissance community, make sure that you follow it to guarantee an enjoyable experience for every member of the community.
Renaissance Benchmark Suite is built via SBT and is written in both Java and Scala.
Several tips on how to setup your IDE are in documentation/ide.md.
To build the suite and create the so-called fat JAR (or super JAR), you need to have
a JDK (at least version 11) installed and run the sbt build tool as follows:
tools/sbt/bin/sbt renaissancePackage
This will retrieve all the dependencies, compile all the benchmark projects and the harness,
bundle the JARs and create the final JAR under the target directory.
Contribution to Renaissance Benchmark Suite is not only about adding new benchmarks. We welcome authors of new plugins, testers of not-so-common platforms and we are open to general discussion about improving the suite too.
In the open-source spirit, anyone can propose an additional benchmark by opening a pull request. The code is then inspected by the community -- typically, the suite maintainers are involved in the review, but anybody else is free join the discussion. During the discussion, the reviewers suggest the ways in which the benchmark could be improved.
Before submitting a pull request, it is recommendable to open an issue first, and discuss the benchmark proposal with the maintainers.
More details about the actual source code organization and guides can be found in the following files.
- Overall suite design:
documentation/design.md - Adding a new benchmark:
documentation/adding-benchmark.md - Adding a plugin:
documentation/plugins.md
Here is some of the quality criteria that a new benchmark should satisfy:
- Stylistically conformant: the benchmark code must conform to existing formatting and stylistic standards in the suite.
- Meaningful: it must represent a meaningful workload that is either frequently executed, or it consists of code patterns and coding styles that are desired or arguably preferable, or it encodes some important algorithm, a data structure or a framework, or is significant in some other important way. If the benchmark is automatically generated by some tool, then it must be accompanied with a detailed explanation of why the workload is useful.
- Observable: it must constitute a run that is observable and measurable. For example, the run should last sufficiently long so that it is possible to accurately measure its running time, gather profiling information or observe a performance effect. Typically, this means that the benchmark duration should be between 0.5 and 10 seconds.
- Deterministic: the performance effects of the benchmark should be reproducible. Even if the benchmark consists of, for example, concurrent code that is inherently non-deterministic, the benchmark should behave relatively deterministically in terms of the number of threads that it creates, the objects it allocates, and the total amount of computation that it needs to do. For example, each of the benchmark repetitions should have a relatively stable running time on major JVMs.
- New: it must contain some new functionality that is not yet reflected in any of the existing benchmarks. If there is significant overlap with an existing benchmark, then it should be explained why the new benchmark is necessary.
- Self-contained: it must be runnable within a single JVM instance, without any additional software installation, operating system functionality, operating system services, other running processes, online services, networked deployments or similar. The JVM installation should be sufficient to run all the code of the benchmark. For example, if the benchmark is exercising Apache Spark, then the workers should run in the local mode, and if the benchmark targets a database, then it should be embedded. The benefit is that the performance effects of the benchmark are easy to measure, and the code is reproducible everywhere.
- Open-source: the benchmark must consist of open-source code, with well-defined licenses.
While the open-source process is designed to accept contributions on an ongoing basis, we expect that this benchmark suite will grow considerably over the course of time. We will therefore regularly release snapshots of this suite, which will be readily available. These will be known as minor releases.
Although we will strive to have high-quality, meaningful benchmarks, it will be necessary to proliferate the most important ones, and publish them as major releases. This way, researchers and developers will be able to test their software against those benchmarks that were deemed most relevant. A major release will still include all the benchmarks in the suite, but the list of highlighted benchmarks might evolve between major releases.
Once a year, a committee convenes and discusses which important benchmarks were contributed since the last release, and should become highlighted in the next major release. The committee consists of members from various universities and companies, who are involved in research and development in virtual machine, compiler, memory management, static and dynamic analysis, and performance engineering.
The committee goes over the benchmark list, and inspect the new ones since the last release. The members can recommend specific benchmarks for inclusion in the highlighted list, and can present their arguments about why those benchmarks should be included. In addition, the members can recommend the exclusion of some benchmarks. The committee members then vote, and the majority is the basis for a decision.
The new major release is then bundled and the binaries are made available publicly.
The current members of the committee are:
- Walter Binder, Universita della Svizzera italiana
- Steve Blackburn, Australian National University
- Lubomir Bulej, Charles University
- Gilles Duboscq, Oracle Labs
- Francois Farquet, Oracle Labs
- Vojtech Horky, Charles University
- David Leopoldseder, Johannes Kepler University Linz
- Guillaume Martres, Ecole Polytechnique Federale de Lausanne
- Aleksandar Prokopec, Oracle Labs
- Andrea Rosa, Universita della Svizzera italiana
- Denys Shabalin, Ecole Polytechnique Federale de Lausanne
- Petr Tuma, Charles University
- Alex Villazon, Universidad Privada Boliviana