You are reading immutable version 39. The current guide may be newer.

Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Comprehensive Assessment

Paper concept map

Interactive study view

Paper concept map

Paper evidence

Follow how prerequisites and the paper’s main ideas connect in teaching order.

Prerequisites

  • Basic understanding of deep learning architectures
  • Familiarity with sequence-to-sequence models and recurrent neural networks

Concepts

  • Explain the core motivation for replacing recurrent neural networks with self-attention
  • Derive the Scaled Dot-Product Attention formula and explain the scaling factor
  • Analyze the mechanism and benefits of Multi-Head Attention
  • Describe the role and formulation of sinusoidal positional encodings

Outcomes

  • Evaluate the empirical performance of the Transformer on translation and parsing tasks
  1. Basic understanding of deep learning architectures requires Explain the core motivation for replacing recurrent neural networks with self-attention
  2. Familiarity with sequence-to-sequence models and recurrent neural networks requires Explain the core motivation for replacing recurrent neural networks with self-attention
  3. Explain the core motivation for replacing recurrent neural networks with self-attention influences Derive the Scaled Dot-Product Attention formula and explain the scaling factor
  4. Derive the Scaled Dot-Product Attention formula and explain the scaling factor influences Analyze the mechanism and benefits of Multi-Head Attention
  5. Analyze the mechanism and benefits of Multi-Head Attention influences Describe the role and formulation of sinusoidal positional encodings
  6. Describe the role and formulation of sinusoidal positional encodings influences Evaluate the empirical performance of the Transformer on translation and parsing tasks