A Survey of Transformers (2021)

[DataLama]

📜 A Survey of Transformers

⚡ 한줄요약

2021년 6월 기준으로 정리한 transformer 아키텍쳐에 대한 서베이 논문.

🏷️ Abstract

Transformers have achieved great success in many artificial intelligence fields, such as natural language
processing, computer vision, and audio processing. Therefore, it is natural to attract lots of interest from
academic and industry researchers. Up to the present, a great variety of Transformer variants (a.k.a. X-formers)
have been proposed, however, a systematic and comprehensive literature review on these Transformer variants
is still missing. In this survey, we provide a comprehensive review of various X-formers. We first briefly
introduce the vanilla Transformer and then propose a new taxonomy of X-formers. Next, we introduce the
various X-formers from three perspectives: architectural modification, pre-training, and applications. Finally,
we outline some potential directions for future research.

🗄️ References

• 논문
• DSBA 연구실 요약

[DataLama]

1. Introduction

• Transformer 모델이 NLP, CV, Audio 도메인에서 Seq2Seq 구조를 바탕으로 많은 improvement를 가져옴.
• Transformer 출현 이후로 다음과 같은 세가지 관점에서 변형 모델이 활발히 연구됨.
  • Model Efficiency
    • Quadratic Complexity of attention map
  • Model Generalization
    • Limitation: Inductive bias와 관련하여 최소한의 가정을 하고 있기 때문에, 적은 데이터셋에 쉽게 overfit이 되는 한계가 있음.
    • Improvement : structural bias, regularization, SSL
  • Model Adaptation
    • Specific한 downstream task에 적용하기 위한 변형.

[DataLama]

2. Background

Vanila Transformer에 대한 설명. (All you need is attention 2017)

transformer architecture에 대한 설명

모델 복잡도

다른 네트워크와 비교