🍒 Cherry in PyTorch

Reinforcement Learning & other bedtime stories in PyTorch. This repository is pytorch implementation of the Intro RL course & Advanced RL course. The name cherry is derived from Yann Lecun's statement regarding RL being the cherry on the ML cake. For training/playing the agent within conda env, please install the external dependencies. Not to be confused with a package of similar name.

Table of Contents

• Installation
  • Conda or Docker
• Sample usage
  • Cartpole-v0
• Dependencies (Conda only)
  • Vizdoom, ALE, Baselines
• Architectures & Agents
• Environments
• Design principle
• Reproducibility

💻 Installation

Tested with ubuntu:18.04 only

🐍 Conda

# After installed external dependencies
conda create -n cherry python=3.6
conda activate cherry
pip install -r requirements.txt
pip install -e .

🐳 Docker

# CPU
docker pull moabitcoin/cherry-pytorch-cpu
docker run moabitcoin/cherry-pytorch-cpu cherry --help
# GPU
docker pull moabitcoin/cherry-pytorch-gpu
docker run moabitcoin/cherry-pytorch-gpu cherry --help

🐳 Docker (build from scratch)

# CPU
make install dockerfile=Dockerfile.cpu dockerimage=moabitcoin/cherry-pytorch-cpu
make run dockerimage=moabitcoin/cherry-pytorch-cpu
# GPU
make install dockerfile=Dockerfile.gpu dockerimage=moabitcoin/cherry-pytorch-gpu
make gpu dockerimage=moabitcoin/cherry-pytorch-gpu

🎉 Usage

# Inside the virtual environment / Docker
cherry --help
cherry train --help

usage: cherry train [-h] -c CONFIG_FILE [-d {gpu,cpu}] [-l {info,debug}]

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIG_FILE, --config_file CONFIG_FILE
                        Path to Config file (default: None)
  -d {gpu,cpu}          Device to run the train/test (default: gpu)
  -l {info,debug}, --log {info,debug}
                        Set verbosity for the logger (default: info)

👀 Example (Cartpole)

Train

cherry train -c configs/control.yaml -d cpu

Play

# <model_dest> in configs/control.yaml
cherry play -c <model_dest>/control-<commit-gitsha>.yaml -d cpu -m <model_dest>/agent-final-<commit-gitsha>.pth

Visualise

# <state_dest> in configs/control.yaml
vlc <state_dest>

Architectures

We support 4 feedforward architectures within cherry. We plan to expand the list of architectures to include Recurrent/Transformer/Memory architectures. If your personal model flavour is missing, please open an issue with links to architecture details.

Agents

We support 4 Agents within cherry. We plan to expand the list of agents to include TRPO/PPO. Please feel free to make an agent request by opening an issue with useful links to publication(s)/existing implementation.

Environments

We support 4 environments within cherry. This list is planned to be expanded to include Robotics and other 3rd party environments.

👬 Dependencies

ViZDoom

Install dependencies as outlined here

sudo apt-get install libcanberra-gtk-module
pip install vizdoom
# If on a headless machine
export DISPLAY=:0

👾 Arcade Learning Environment

https://github.com/openai/gym/blob/master/docs/environments.md#atari

📈 OpenAI RL baselines

Needed for Deepmind styled pre-processing wrappers. Needs TF>=1.14 🤦‍♀️

pip install tensorflow_gpu==1.14
https://github.com/openai/baselines/tree/master#installation

📺 Remote Render

https://github.com/openai/gym/tree/0.15.4#rendering-on-a-server
xvfb-run -s "-screen 0 1400x900x24" bash