Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Introduction to the Transformer Architecture

Sequential Computation Bottleneck in Recurrent Models

The Sequential Bottleneck of Recurrent Architectures

Sources

S1.p2.3

Recurrent models typically factor computation along the symbol positions of the input and output sequences. Aligning the positions to steps in computation time, they generate a sequence of hidden states hth_{t}, as a function of the previous hidden state ht−1h_{t-1} and the input for position tt. This inherently sequential nature precludes parallelization within training examples, which becomes critical at longer sequence lengths, as memory constraints limit batching across examples. Recent work has achieved significant improvements in computational efficiency through factorization tricks [21] and conditional computation [32], while also improving model performance in case of the latter. The fundamental constraint of sequential computation, however, remains.

Traditional sequence transduction models rely on recurrent neural networks (RNNs) that process tokens sequentially. To compute the hidden state $h_t$ at step $t$ , the model must wait for the computation of the previous hidden state $h_{t-1}$ and combine it with the current input $x_t$ . This sequential dependency prevents parallelization within a single training example, leading to significant computational bottlenecks as sequence lengths grow.

Sources

S1.p2.3

Recurrent models typically factor computation along the symbol positions of the input and output sequences. Aligning the positions to steps in computation time, they generate a sequence of hidden states hth_{t}, as a function of the previous hidden state ht−1h_{t-1} and the input for position tt. This inherently sequential nature precludes parallelization within training examples, which becomes critical at longer sequence lengths, as memory constraints limit batching across examples. Recent work has achieved significant improvements in computational efficiency through factorization tricks [21] and conditional computation [32], while also improving model performance in case of the latter. The fundamental constraint of sequential computation, however, remains.

Deep dive

The Transformer Solution: By replacing recurrence entirely with attention mechanisms, the Transformer allows global dependencies to be drawn in parallel across all positions in a sequence, bypassing the sequential step-by-step constraint.

Sources

S1.p4.1

In this work we propose the Transformer, a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output. The Transformer allows for significantly more parallelization and can reach a new state of the art in translation quality after being trained for as little as twelve hours on eight P100 GPUs.