T5-transformer

Source: Google AI Blog^[1]

About The Project

Study of the methodology proposed in the paper Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.

In this project one of the T5 models - precisely T5-small - is fine-tuned over a fraction of the tasks performed in the paper. In addition, to further extend this work it was tested the adapter layers variant for fine-tuning.

More details about the project in the presentation.

Built With

• 🤗 Transformers
• PyTorch
• Adapter Transformers

Usage

In the repo there are two notebooks, one for "standard" fine-tuning and one for fine-tuning with adapters. To facilitate reading, the notebooks follow the pipeline for the question-answering task. Despite this, the implementations are totally generalizable to all types of tasks.

1. Execute t5-fine-tune.ipynb to fine-tune the model. In order to change task, it might be necessary to make small adjustments in the pre-processing setup.
2. Execute t5-fine-tune-adapters.ipynb to fine-tune the model with the adapters. To manipulate the inner dimensionality of the adapter layer it is sufficient to change properly the value of the reduction factor.

Acknowledgments

1. The image has been taken from this Google AI Blog.