Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Comparing Self-Attention, Recurrent, and Convolutional Layers

Limitations of Recurrent Neural Networks in Sequence Modeling

In sequence modeling, recurrent neural networks process tokens sequentially, which creates a fundamental computational bottleneck. Because the computation of hidden state $h_t$ depends on $h_{t-1}$ , operations cannot be easily parallelized across the sequence length $n$ . This results in $O(n)$ sequential operations, as opposed to $O(1)$ for self-attention layers.

Sources

S4.T1

Table 1: Maximum path lengths, per-layer complexity and minimum number of sequential operations for different layer types. nn is the sequence length, dd is the representation dimension, kk is the kernel size of convolutions and rr the size of the neighborhood in restricted self-attention. Layer Type Complexity per Layer Sequential Maximum Path Length Operations Self-Attention O(n2⋅d)O(n^{2}\cdot d) O(1)O(1) O(1)O(1) Recurrent O(n⋅d2)O(n\cdot d^{2}) O(n)O(n) O(n)O(n) Convolutional O(k⋅n⋅d2)O(k\cdot n\cdot d^{2}) O(1)O(1) O(logk(n))O(log_{k}(n)) Self-Attention (restricted) O(r⋅n⋅d)O(r\cdot n\cdot d) O(1)O(1) O(n/r)O(n/r)