Empirical analysis of PGA-MAP-Elites for Neuroevolution in Stochastic Domains

This repository contains the code associated with Empirical analysis of PGA-MAP-Elites for Neuroevolution in Stochastic Domains.

It proposes an implementation of several algorithms compared within the paper:

• MAP-Elites algorithm from Illuminating search spaces by mapping elites.
• TD3 proposed in Addressing Function Approximation Error in Actor-Critic Methods.
• PGA-MAP-Elites first introduced in Policy Gradient Assisted MAP-Elites.
• CMA-MAP-Elites from Covariance Matrix Adaptation for the Rapid Illumination of Behavior Space.
• Deep-grid-MAP-Elites introduced in Fast and stable MAP-Elites in noisy domains using deep grids.
• MAP-Elites-sampling as used in previous works such as Hierarchical Behavioral Repertoires with Unsupervised Descriptors.

This code compares these approaches on the QDgym_extended tasks based on the QDgym tasks. It is based on the original repository of PGA-MAP-Elites, merging the repository of TD3 with the PyMAP-Elites implementation of MAP-Elites. We provide the data used to generate the paper results in zip files in 2022_results. We provide an executable precompiled Singularity container here.

Libraries and dependencies

The implementation of all tasks and algorithms is in Python 3.8. It requires the standard Python 3.8 libraries, Numpy, Scikit-learn for the implementation of the CVT archives, and PyRibs for CMA-MAP-Elites. All algorithms learn Deep Neural Network controllers, requiring the PyTorch library, and all the tasks are implemented in the QDgym_extended library that relies on Gym and Pybullet. Furthermore, the analysis of the results is based on Panda, Matplotlib and Seaborn libraries.

We also propose a containerised version of our environment to replicate our experiment and results in Singularity. We provide an executable precompiled Singularity container for this repository here.

Structure

• run_experiment.py is the main file, used to run any experiments.
• run_plots.py allows for analysing the results of a run and generating the corresponding graphs.
• configure_experiment contains one config file to run each algorithm with its default parameters.
• CVT saves the CVT archives to avoid recomputing them at each run.
• src contains the definition of all algorithms based on the structure of PyMAP-Elites.
  • initialisation.py is called first to initialise the chosen algorithm.
  • mapelites.py runs the main loop of the algorithm.
  • mapping contains all the classes and functions needed to handle the main component of MAP-Elites: the archive.
  • learning contains all the classes for Deep Neural Networks objects: the controllers for all algorithms and the critics and replay-buffers for Deep Reinforcement Learning approaches.
  • process defines the processes parallel to the main loop: the critic training and the parallelisation of the evaluations.
  • variation allows defining the MAP-Elites mutations and the policy-gradient mutations.
  • metrics contains the definition of all metrics used in the paper.
  • analysis defines all the functions needed to analyse the data output by a run.
• singularity contains all files needed to compile the Singularity container for this experiment. You can find a precompiled container for this repository here.
• 2022_results contains the data used to generate the paper results.

Execution from source

Dependencies to run form source

Running this code from source requires Python 3.8, and the libraries given in requirements.txt (Warning: preferably use a virtual environment for this specific project, to avoid breaking the dependencies of your other projects). In Ubuntu, installing the dependencies can be done using the following command:

pip install -r requirements.txt

Troubleshooting

Depending on your setup, you might get an error when installing scikit-learn from the requirements.txt, this is due to pip trying to install the package in the wrong order. You can enforce it by first running:

pip3 install numpy==1.19.5 Cython==0.29.21 GPUtil==1.4.0 psutil==5.9.0

You can then run the requirements.txt installation safely.

Run from source

The results from the experiments can be reproduced by running run_experiment.py. This script takes a range of arguments and the configure_experiment folder contains .txt configuration files with default arguemnt values for each algorithms. Configuration files can be passed using the --config_file argument. For example, to run PGA-MAP-Elites with the parameters from the paper:

python3 run_experiment.py --config_file configure_experiment/pga_config.txt

By default, this would run the QDWalker2DBulletEnv-v0 environment. To change it, for example, to QDAntBulletEnv-v0, use the --env argument. To run a short-time test, for example, for 1000 evaluations instead of 1000000, use the --max_evals argument:

python3 run_experiment.py --config_file configure_experiment/pga_config.txt --env QDAntBulletEnv-v0 --max_evals 1000

The output of the run will be saved in the location specified by the --save_path argument. Other arguments are detailed in run_experiment.py. Be careful that the seed is fixed for a given run and needs to be passed as an argument using --seed, the results of the paper used a random different seed for each run.

Analysis from source

The output of a run contains .csv, .dat and .pickle files that can be analysed and plotted using run_plots.py. To execute it, use:

python3 run_plots.py --save_path *path_to_stats_files* --results_path *path_to_save_graphs* --max_evals *max_evals_of_run*

--save_pathspecifies where the path to the files output by the run, --results_path specifies where to save the resulting graphs and --max_evals allows selecting which archive should be plotted as the final archive. If there is no archive corresponding to this value, no archive plots will be displayed. Other arguments are listed in run_plots.py. We provide the data used to generate the paper results in zip files in 2022_results.

run_plots.py also allows visualising the policies in the PyBullet environment using the --visualisation argument.

Execution from the final Singularity container image

Run with an existing container

The results of the paper can be reproduced by running the Singularity container of the experiment, for example, the precompiled container provided here. A container image can be executed directly: ./*image_name*. This will re-generate the results for PGA-MAP-Elites on the QDWalker task with the parameters from the paper. Arguments, as detailed in previous sections, can be specified when executing the container. For example, to run PGA-MAP-Elites for 1000 evaluations on the QDAnt task using the provided image, one can execute the command:

./pga_map_elites.sif --env QDAntBulletEnv-v0 --max_evals 1000

Unless otherwise specified, the results of the execution are solved in a results_pga-map-elites folder, outside of the image. Executing the container will also run the analysis of these results and generate the corresponding graphs as .svgand .pngfiles.

Analysis with an existing container

The Singularity container also provides a separateAnalysis app to run the analysis only. It can be called using:

singularity run --app Analysis *image_name* *results_folder_name* *additional_analysis_parameters*

Compiling a new Singularity container

The singularity folder provides all the building blocks to recompile a new Singularity container:

• singularity.def contains the "recipe" to build the container: it starts from a Ubuntu container, updates a existing libraries and installs the package in requirements.txt.
• start_container allows to build a sandbox container, for development. It can be executed with ./start_container.
• build_final_image recompiles a final container similar to the one provided here. It can be executed with ./build_final_image.