Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Comparing Self-Attention, Recurrent, and Convolutional Layers

Restricted Self-Attention for Long Sequences

For extremely long sequences, self-attention can be restricted to a local neighborhood of size $r$ centered around each output position. This restricted self-attention reduces complexity per layer to $O(r \cdot n \cdot d)$ while increasing the maximum path length to $O(n/r)$ .

Sources

S4.p4.5

As noted in Table 1, a self-attention layer connects all positions with a constant number of sequentially executed operations, whereas a recurrent layer requires O(n)O(n) sequential operations. In terms of computational complexity, self-attention layers are faster than recurrent layers when the sequence length nn is smaller than the representation dimensionality dd, which is most often the case with sentence representations used by state-of-the-art models in machine translations, such as word-piece [38] and byte-pair [31] representations. To improve computational performance for tasks involving very long sequences, self-attention could be restricted to considering only a neighborhood of size rr in the input sequence centered around the respective output position. This would increase the maximum path length to O(n/r)O(n/r). We plan to investigate this approach further in future work.