Study the paper
Attention Is All You Need
Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.
Why attention replaces recurrence
Why replace recurrence?
Why replace recurrence?
The paper introduces an architecture that replaces recurrent sequence processing with attention-driven computation.
Sources
S1.p1.1
Recurrent neural networks, long short-term memory [13] and gated recurrent [7] neural networks in particular, have been firmly established as state of the art approaches in sequence modeling and transduction problems such as language modeling and machine translation [35, 2, 5]. Numerous efforts have since continued to push the boundaries of recurrent language models and encoder-decoder architectures [38, 24, 15].
Deep dive
The supplied excerpt establishes the replacement claim; it does not by itself quantify every computational tradeoff.
Sources
section
1 Introduction
S1.p1.1
Recurrent neural networks, long short-term memory [13] and gated recurrent [7] neural networks in particular, have been firmly established as state of the art approaches in sequence modeling and transduction problems such as language modeling and machine translation [35, 2, 5]. Numerous efforts have since continued to push the boundaries of recurrent language models and encoder-decoder architectures [38, 24, 15].