Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Introduction to the Transformer Architecture

Limitations of Recurrent and Convolutional Architectures

The Sequential Bottleneck of Recurrent Architectures

Sources

S1.p2.3

Recurrent models typically factor computation along the symbol positions of the input and output sequences. Aligning the positions to steps in computation time, they generate a sequence of hidden states hth_{t}, as a function of the previous hidden state ht−1h_{t-1} and the input for position tt. This inherently sequential nature precludes parallelization within training examples, which becomes critical at longer sequence lengths, as memory constraints limit batching across examples. Recent work has achieved significant improvements in computational efficiency through factorization tricks [21] and conditional computation [32], while also improving model performance in case of the latter. The fundamental constraint of sequential computation, however, remains.

In sequence transduction tasks, recurrent neural networks (RNNs) have traditionally been the dominant architecture. However, they suffer from a fundamental computational bottleneck: they process sequences step-by-step. Specifically, an RNN computes a sequence of hidden states $h_t$ as a function of the current input at position $t$ and the previous hidden state $h_{t-1}$ . This sequential dependency means that the computation for step $t$ cannot begin until the computation for step $t-1$ is complete, preventing parallelization across the sequence length during training.

Sources

S1.p2.3

Recurrent models typically factor computation along the symbol positions of the input and output sequences. Aligning the positions to steps in computation time, they generate a sequence of hidden states hth_{t}, as a function of the previous hidden state ht−1h_{t-1} and the input for position tt. This inherently sequential nature precludes parallelization within training examples, which becomes critical at longer sequence lengths, as memory constraints limit batching across examples. Recent work has achieved significant improvements in computational efficiency through factorization tricks [21] and conditional computation [32], while also improving model performance in case of the latter. The fundamental constraint of sequential computation, however, remains.