Study the paper
Attention Is All You Need
Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.
Introduction to the Transformer Architecture
At a glance
At a glance
Introduction to the Transformer Architecture
Sources
section
1 Introduction
Traditional sequence transduction models rely heavily on recurrent neural networks (RNNs), which compute hidden states sequentially over time steps. This sequential constraint prevents parallelization within training examples, posing a major bottleneck for long sequences.
Sources
S1.p2.3
Recurrent models typically factor computation along the symbol positions of the input and output sequences. Aligning the positions to steps in computation time, they generate a sequence of hidden states hth_{t}, as a function of the previous hidden state ht−1h_{t-1} and the input for position tt. This inherently sequential nature precludes parallelization within training examples, which becomes critical at longer sequence lengths, as memory constraints limit batching across examples. Recent work has achieved significant improvements in computational efficiency through factorization tricks [21] and conditional computation [32], while also improving model performance in case of the latter. The fundamental constraint of sequential computation, however, remains.
To address this limitation, the Transformer architecture completely eschews recurrence. Instead, it relies entirely on an attention mechanism to model global dependencies between inputs and outputs, enabling significantly higher parallelization during training.
Sources
S1.p4.1
In this work we propose the Transformer, a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output. The Transformer allows for significantly more parallelization and can reach a new state of the art in translation quality after being trained for as little as twelve hours on eight P100 GPUs.