Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Introduction to the Transformer Architecture

Motivation for the Transformer

Recurrent neural networks (RNNs) process sequences sequentially, computing a hidden state $h_t$ as a function of the previous hidden state $h_{t-1}$ and the current input. This step-by-step alignment prevents parallelization within a single training example, leading to severe computational bottlenecks for long sequences.

Sources

S1.p2.3

Recurrent models typically factor computation along the symbol positions of the input and output sequences. Aligning the positions to steps in computation time, they generate a sequence of hidden states hth_{t}, as a function of the previous hidden state ht−1h_{t-1} and the input for position tt. This inherently sequential nature precludes parallelization within training examples, which becomes critical at longer sequence lengths, as memory constraints limit batching across examples. Recent work has achieved significant improvements in computational efficiency through factorization tricks [21] and conditional computation [32], while also improving model performance in case of the latter. The fundamental constraint of sequential computation, however, remains.

The Transformer architecture addresses this limitation by completely removing recurrence and convolution. Instead, it relies entirely on self-attention mechanisms to capture global dependencies across the entire sequence in parallel.

Sources

S1.p4.1

In this work we propose the Transformer, a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output. The Transformer allows for significantly more parallelization and can reach a new state of the art in translation quality after being trained for as little as twelve hours on eight P100 GPUs.