Deep Q-Network Agent For Snake Game

This repository holds code and reports for an applied reinforcement learning research project I carried out in Fall 2021. The objective of the project was the Python implementation of a simple Deep Q-Network (DQN) agent for the classic game of Snake using the PyTorch machine learning framework and the Pygame library.

Several DQN agents were trained on different combinations of environment and incentives – borders, obstacles, rewards, and penalties. Furthermore, agents trained on one combination of environment and incentives were tested on other combinations of environment and incentives to investigate how much the agent's training generalizes when faced with novel situations.

The agents were trained for 500 games, and each test also lasted for 500 games. The results of the training and testing ablation studies conducted are presented and discussed in the RL Project_ Deep Q-Network Agent Report PDF file, and summarized in the RL Project_ Deep Q-Network Agent Presentation PDF file.

The code implementation was inspired by a similar open-source project ^{1 2} by Patrick Loeber. Do check out his YouTube channel and website for more informative Machine Learning, Computer Vision, and Data Science content.

Code Implementation in Python, PyTorch & Pygame

The Python code implementation is split into four modules. The details, summarized below, can be found in the RL Project_ Deep Q-Network Agent Report PDF file.

1. _dqnAINetwork.py:
   This helper module contains the PyTorch implementation of the neural network (NN) models and architectures. This module also contains the Q-Learning algorithm for calculating the loss across game states and training the agent.

2. _dqnSnakeEnvironment.py:
   This helper module contains the Pygame implementation of the game environment (borders, obstacles, food and the snake) within the playing area. This module outputs information needed to abstract the game state and inputs information to determine the next move of the agent and updates the user interface.

3. _dqnAIAgent.py:
   This helper module implements the snake game agent by building upon and connecting the _dqnAINetwork.py and _dqnSnakeEnvironment.py helper modules. It abstracts the game state variables from _dqnSnakeEnvironment.py and passes the state variables to _dqnAINetwork.py to determine the agent's next move.

4. dqnSnakeMain.py:
   This is the main module for running the Snake Game Deep Q-Network code implementation. It contains three functions – which depend on the _dqnSnakeEnvironment.py and _dqnAIAgent.py helper modules – viz:

• AITrainer(): Trains an agent model for a number of games within a specified environment .
• AITester(): Loads a trained agent model for testing in a specified environment.
• userMode(): Sets up a game with a specified environment with the user's keyboard input determining the snake's movement.

The ./models folder contains some trained agent models. The suffixes Mxxx represents the maximum score (in one game) achieved by the agent during training across 500 games.

---

Training Demo

AITrainer(max_games=500, borders="closed", obstacles=0, file_name="demo_border=closed_obstacles=0.pth")

Testing Demo

AITester(max_games=500, borders="closed", obstacles=0, model_file="demo_border=closed_obstacles=0.pth")

Footnotes

1. L. Patrick, Teach AI To Play Snake! Reinforcement Learning With PyTorch and Pygame., 2021. ↩

2. L. Patrick, Teach AI To Play Snake - Practical Reinforcement Learning With PyTorch And Pygame, Python Engineer. [Accessed: 12-Oct-2021]. ↩