dhok_directedtests

[monikadhok]

[ctreude]

The authors introduce an approach to automatically generate tests to detect redundant loop traversals in Java, aiming to effectively locate potential performance issues.

The artifact works as advertised. Everything is easy to install and all the instructions are straightforward. The authors have done an excellent job documenting everything at an appropriate level of detail. The artifact is very useful since it allows the generation of tests on the fly for any source code. I would encourage the authors to also think of making their work available as a web service or an IDE plugin.

I'm happy to recommend the inclusion of this dataset in the online repository, the social media campaign and the award session.

This artifact consists of a technique that addresses the problem of automatically generating tests to detect redundant loop traversals. According to the authors, redundant loop traversal is one of the primary sources for performance issues in many Java libraries.

Insightful

The idea of the paper is interesting, timely, and relevant for this FSE 2016 artifact track.

The paper is well motivated and the problem is clearly stated.

Useful

The authors leverage dynamic analysis to generate targeted tests that expose loop inefficiencies. The aim is to overcome shortcomings of previous works that heavily rely on manual work and analysis of classes, which is tedious, impractical and time consuming.

The technique seems to be cost effective.

Usable

The authors adhere to the FSE artifact track guidelines and particularly the three required dimensions: insightfulness, usefulness, and usability.

The given artifact consists of a script to execute all the benchmarks.

Details related to the artifact package are provided including the hardware requirements, installation, and experimental setup.

The artifact is available online and provided in the form of compressed tar.gz file.

Additional Comments:

In section 1.2, the authors claim that their technique enhances the effectiveness of the state of the art approaches to detect similar problems. Did you run a comparative study to draw such a conclusion and—or confirm these findings? If yes, please add a reference to the paper/study.

What about this effectiveness? How it was measured? And what about the results in terms of numbers? Could you please add information about how accurate is the suggested technique?

The authors state that their technique has detected a number of bugs across several Java libraries. And that the reported bugs are confirmed and fixed by developers. Could you please add information about this? Plus, a reference to the work?

You mentioned “benchmarks” or “all benchmarks” across the paper. Please add a note in the paper about what these benchmarks are/consists of.

Minor:

In section 1.2: please replace “targetted tests” by “targeted tests”.

Overall, I believe this artifact can be of great value to the SE community especially the one working in the testing area. And I therefore mark it as an accept. I evaluate it as ‘Maybe Gold’.

[mkramanathan]

@Latifa-Guerrouj -- Thanks for your review.

1. For the concerns mentioned under additional comments, is it a review of the paper or that of the artifact?
2. Aren't these details already available in the Experimental Section (Sec 5) of the paper?
3. Is the suggestion to include these details once again in the writeup of the artifact?
4. Was there any problem in reproducing the results mentioned in the paper?

[agurfinkel]

The artifact is a package to reproduce the experiments in the
corresponding paper. A detailed tutorial to reproduce the experiments
is included. The artifact also includes instructions on configuring a
virtual machine.

I found the instructions easy to follow and could reproduce the
experiments.

The artifact is necessary to support the empirical results used in the
paper and might be useful for researchers interested in conducting
additional experiments.