Bug Bench

Benchmark

benchmark
|--- PROGRAM1
    `--- VER1
    |    `--- label.json
    |    `--- Dockerfile
    |    `--- build.sh
    `--- VER2
    |    `--- label.json
    |    `--- Dockerfile
    |    `--- build.sh
    ...

Bug Label

See LABEL.md.

Docker

• Base image: prosyslab/bug-bench-base in Dockerhub built from docker/Dockerfile

/
|-- src/
|  `-- PROGRAM/
|  `-- build.sh
|-- smake/
|-- infer/   # shall be mounted
|-- codeql/  # shall be mounted
|-- out/
   `-- smake-out/
   `-- infer-out/
   `-- codeql-db/

Building Docker images

• Building a single image

$ bin/build-docker.sh benchmark/[program]/[version]

• Building all images including base

$ bin/build-docker.sh all

How to build with respect to desired target?

$ $BUILD [ sparrow | infer | codeql ]

For example, to build target for Sparrow, run $BUILD sparrow. Then, the built target along with either Sparrow or Infer will be located at /out/*.

How to mount infer or codeql?

$ docker run -it -v PATH/TO/INFER/DIR/:/infer REPO:TAG

or

$ docker run -it -v PATH/TO/CODEQL/DIR/:/codeql REPO:TAG

FYI, look out for -v option in docker run --help.

How to upload repository to prosyslab-warehouse?

First, you have to install and authenticate github-cli on your local machine. To install, see here.

Then, authenticate with your github account:

$ gh auth login

Once auth is done, run the command below to upload source code of the benchmark:

$ bin/add_repo_to_remote.sh [repo name] [url of tar file]

Example

$ bin/add_repo_to_remote.sh prosyslab-warehouse/shntool-3.0.5 http://shnutils.freeshell.org/shntool/dist/src/shntool-3.0.5.tar.gz

Contributing

See CONTRIBUTING.md.