POLICEd-RL: Learning to Provably Satisfy High Relative Degree Constraints for Black-Box Systems

Overview

Repository containing code to implement high relative degree POLICEd RL presented at CDC 2024. The objective of POLICEd RL is to guarantee the satisfaction of an affine hard constraint of high relative degree when learning a policy in closed-loop with a black-box deterministic environment. The algorithm enforces a repulsive buffer in front of the constraint preventing trajectories to approach and violate this constraint. To analytically verify constraint satisfaction, the policy is made affine in that repulsive buffer using the POLICE algorithm.

POLICEd RL guarantees that this space shuttle will never land with a vertical velocity higher than 6ft/s thanks to the green repulsive buffer.

We provide the code for our implementation of POLICEd RL on several systems:

• the Gymnasium Inverted Pendulum
• a space shuttle landing

Organization

• code contains the project source code,
• docs contains the code for our website.

Credit

The following repositories have been instrumental from both an algorithm and software architecture perspective in the development of this project:

• RandallBalestriero/POLICE
• sfujim/TD3
• labicon/POLICEd RL