Autoscale

This repo contains the scripts for running the Autoscale benchmark generation tool as described in the ICAPS 2021 paper "Automatic Instance Generation for Classical Planning" by Álvaro Torralba, Jendrik Seipp and Silvan Sievers.

See https://github.com/AI-Planning/autoscale-benchmarks for the generated benchmarks.

• Directory "autoscale" contains the main scripts for optimizing and selecting sequences.

• Directory "experiments" contains the main bulk of the experiment scripts for all phases of the instance generation approach described in the paper. See usage instructions below.

• Directory "pddl-generators" contains all generators of IPC domains that were used for this paper (and some additional ones).

• Directory "planners" contains source code as well as Singularity recipe files for all planners used in the paper. It also contains two scripts for building all the corresponding Singularity images.

• Directory "logfiles" contains the logs of the select-sequences.py script for the results of the paper.

• Directory "report" contains some additional scripts and Jupyter notebooks used to manually inspect the results of preliminary experiments. They can show some interesting additional data, but they are not intended to be maintained.

Tested versions

• Python 3.7
• Singularity 3.5
• SMAC 0.12.3
• CPLEX 12.10

Installation instructions

We recommend running Autoscale in a Conda environment. The following commands create a Conda environment with the necessary libraries and install the required Python packages (including SMAC).

conda create --name autoscale python=3.7 gxx_linux-64 gcc_linux-64 swig
conda activate autoscale
pip install -r requirements.txt

Singularity

On Debian-based systems, probably the easiest way to install Singularity is to use the Singularity Debian package. If that doesn't work, you might have to compile Singularity from source.

Build planner images:

export DOWNWARD_BENCHMARKS=<path/to/benchmarks/repo>
cd planners
./build-all-images.sh

CPLEX

Install CPLEX 12.10 and add the python/3.7/x86-64_linux/ subfolder to the PYTHONPATH environment variable. By default, CPLEX is installed in /opt/ibm/ILOG/CPLEX_Studio1210, so one has to add /opt/ibm/ILOG/CPLEX_Studio1210/cplex/python/3.7/x86-64_linux/ to the PYTHONPATH.

Usage

Before running the optimization, the following steps are needed:

• Edit domains.py to add the description of any new domain that you want to configure.

• Edit planners.py to add the state-of-the-art planners that are relevant for each domain. The script select-best-planner-per-domain.py can help to automatically select a set of planners per domain from the results of earlier experiments.

Then, the following two steps must be performed:

1. Run optimize-sequences.py to perform the SMAC optimization and generate sequences. The *smac*.py scripts show how to parallelize this step on a computer cluster with Lab.

   Note that the --database option allows to provide the JSON file from a previous SMAC run to re-use all precomputed times for previously tested parameter configurations (see 2020-12-08-A-smac-opt-2014-20h.py for an example).

2. Gather all results from one or multiple SMAC runs into a single properties JSON file. This is done by running the last step of the *smac*.py scripts.

3. Run select-sequences.py to generate a benchmark set. Use the --database option to provide the properties file obtained from step 2. generate-benchmarks-2020-12-10.sh contains an example on how to run the script.

Experiment data

The optimization/results folder contains JSON files with the results of the experiments. The results table in the paper is generated via the scripts report-gather-dataset.py and report-create-table.py.