Optimal Control

This project aims to explore and compare the different methods of solving optimal control problems.

This project will focus on apply different optimal control methods on different types of systems with the goal.

The optimal control methods implemented in this project are categorised based on how much information about the system is available as Planning Methods which require complete information on the system, Model-Based Learning which requires some information on the system, and Model-Free Learning which requires almost no information on the system.

The systems modeled in this project are also categorised based on how much information about the system is available as Deterministic systems, and Stochastic Systems.

The Objective

The objective of this project is to implement an optimal control strategy for different classes of optimal control problems

Table of Contents

• Introduction to Optimal Control Problems
  • Dynamical Systems
    • Properties of dynamical systems
    • Challenges that may arise when modelling a dynamical system
    • Types of dynamical systems
  • Control Theory
    • Basic terminology of control theory
    • Challenges that may arise when controlling a system
    • Methods of control
    • Markov Decision Processes
  • Optimization Problems
    • Types of optimization problems
    • Methods of optimization
    • Optimization in Control Theory
• Optimal Control Methods
  • Analytical/ Planning Methods
    • Variational Calculus
    • Minimum Principle
    • Hamilton-Jacobi-Bellman Equation
  • Learning Methods
    • Dynamic Programming
    • Model-Based Reinforcemnt Learning
    • Model-Free Reinforment Learning
• Implementation of Solutions
  • Case Study 1: Trajectory Optimization
    • System Design
    • Control Implementation
  • Case Study 2: Adaptive Control
    • System Design
    • Control Implementation
  • Case Study 3: Stochastic Control
    • System Design
    • Control Implementation

Code Design

The code for this project will be written in Python with Jupyter Notebook and will be divided into the following sections:

• Optimal Control Methods: This section will contain the implementation of the different optimal control methods.
• System Models: This section will contain the implementation of the different system models.
• Case Studies: This section will contain the implementation of the different case studies.
• Tests: This section will contain the code to test each System Model, and Optimal Control Method
• Tools: This section will contain scripts to automate tasks in the project.

The code is tested with Python 3.12.1 with the following packages:

Package    Version
---------- -------
numpy      1.26.4
pip        24.0
torch      2.3.1

References

Model-based Reinforcement Learning: A Survey.(https://arxiv.org/pdf/2006.16712)

TO DO/ Future Updates

• Complete testing for DDPG Algorithm
• Hyperparametric optimization of DDPG solution of the particle in a field problem
• Complete implementation of the MDP algorithm
• Testing the MDP algorithm
• Use the MDP algorithm to solve the the particle in a field problem
• Benchmark the performance of these algorithms