Jointly Learning to Construct and Control Agents using Reinforcement Learning

This code accompanies the paper Jointly Learning to Construct and Control Agents using Reinforcement Learning.

Arxiv Link: https://arxiv.org/abs/1801.01432

Dependencies

This code is build on top of OpenAI's Roboschool.

To install Roboschool follow the instructions at https://github.com/openai/roboschool

All other dependencies can be installed by calling: pip install -r requirements.txt

Experiments

Experiments are run by calling two scripts: init.py and train.py.

init.py

init.py initializes experiment directories and saves hyperparameters. See init.py for a full list of hyperparameters.

python init.py logdir --hyperparameter value ...

train.py

train.py starts or continues an experiment from the latest checkpoint.

python train.py logdir -s maxseconds --save_freq timesteps_per_checkpoint

Replicating the experiments in the paper

Pretrained models from our experiments are included with the release of this code.

Additionally, for each of the robot morphologies (Hopper, Walker, Ant) and each terrian type (level, inclined), our experiments can be replicated using the following commands

Level Terrain

python init.py logs/hopper_level --robot hopper --terrain flat
python train.py logs/hopper_level

python init.py logs/walker_level --robot walker --terrain flat
python train.py logs/walker_level

python init.py logs/ant_level --robot ant --terrain flat -t 1500000000 --steps_before_robot_update 200000000 --chop_freq 200000000
python train.py logs/ant_level

Inclined Terrain

python init.py logs/hopper_incline --robot hopper --terrain slope
python train.py logs/hopper_incline

python init.py logs/walker_incline --robot walker --terrain slope
python train.py logs/walker_incline

python init.py logs/ant_incline --robot ant --terrain slope -t 1500000000 --steps_before_robot_update 200000000 --chop_freq 200000000
python train.py logs/ant_incline

Evaluation and Visualization

The training script creates Tensorboard summaries in logdir/summaries. These can be visualized by launching Tensorboard:

tensorboard --logdir logdir/summaries

Additionally, two scripts are provided to evaluate and visualize checkpoints: eval.py and viz.py.

eval.py samples robot designs from the design distribution and computes episode statistics averaged over n episodes. Results are saved in logdir/eval

python eval.py logdir -t checkpoint -n nepisodes -s nsamples

viz.py renders or creates videos of the mode of the design distribution. Videos are saved in logdir/videos.

python viz.py logdir -t checkpoint -n nepisodes --save