Attention mechanisms and Transformers

Updates: I'm moving the codebase to a new repository and rewriting it using the latest Keras 3.x version for multi-backend support. I will also port the pretrained weights of the models written here.

• This goal of this repository is to host basic architecture and model traning code associated with the different attention mechanisms and transformer architecture.
• At the moment, I more interested in learning and recreating these new architectures from scratch than full-fledged training. For now, I'll just be training these models on small datasets.

Installation

• Using pip to install from pypi

pip install Attention-and-Transformers

• Using pip to install latest version from github

pip install git+https://github.com/veb-101/Attention-and-Transformers.git

• Local clone and install

git clone https://github.com/veb-101/Attention-and-Transformers.git atf
cd atf
python setup.py install

Example Use

python load_test.py

Attention Mechanisms

# No.	Mechanism	Paper
1	Multi-head Self Attention	Attention is all you need
2	Multi-head Self Attention 2D	MobileViT V1
2	Separable Self Attention	MobileViT V2

Transformer Models

# No.	Models	Paper
1	Vision Transformer	An Image is Worth 16x16 Words:
2	MobileViT-V1	MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer
3	MobileViT-V2	Separable Self-attention for Mobile Vision Transformers
2	MobileViT-V3	MobileViTv3: Mobile-Friendly Vision Transformer