Study the paper
Attention Is All You Need
Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.
21 activities
At a glance
Introduction to the Transformer ArchitectureIntroduction to the Transformer Architecture
LessonLimitations of Recurrent and Convolutional Architectures
Introduction to the Transformer ArchitectureThe Sequential Bottleneck of Recurrent Architectures
LessonScaled Dot-Product Attention
Scaled Dot-Product AttentionMathematical Formulation
LessonScaled Dot-Product Attention
Scaled Dot-Product AttentionThe Scaled Dot-Product Attention mechanism computes attention weights by taking the dot product of queries with keys, scaling them by the square root of the key dimension, applying a softmax function, and using the resulting weights to compute a weighted sum of the values.
VisualHow attention redistributes weight
Scaled Dot-Product AttentionIllustrative attention weights computed from a fixed teaching example; these are not paper-reported values.
VisualExplore attention score scaling
Scaled Dot-Product AttentionMove across fixed illustrative scenarios to see how score scaling changes concentration.
VisualScaled Dot-Product Attention: step by step
Scaled Dot-Product AttentionFollow the existing bounded walkthrough in its intended sequence.
LessonMulti-Head Attention Mechanism
Multi-Head Attention MechanismMulti-Head Attention Mechanism
LessonMulti-Head Attention
Multi-Head Attention MechanismMulti-Head Attention allows the model to jointly attend to information from different representation subspaces at different positions. Instead of performing a single attention function with queries, keys, and values, the queries, keys, and values are projected multiple times with different, learned linear projections.
VisualMulti-Head Attention: step by step
Multi-Head Attention MechanismFollow the existing bounded walkthrough in its intended sequence.
LessonInjecting Order with Positional Encoding
Injecting Order with Positional EncodingInjecting Order with Positional Encodings
LessonSinusoidal Positional Encoding (Even)
Injecting Order with Positional EncodingIn sequence transduction models like the Transformer, which lack recurrence or convolution, positional encodings are added to the input embeddings to inject information about the order of tokens. This equation defines the sinusoidal positional encoding for even dimensions.
VisualSinusoidal Positional Encoding (Even): step by step
Injecting Order with Positional EncodingFollow the existing bounded walkthrough in its intended sequence.
LessonComparing Self-Attention with Recurrence and Convolution
Comparing Self-Attention with Recurrence and ConvolutionComparing Self-Attention, Recurrent, and Convolutional Layers
LessonBLEU scores and training costs on WMT 2014 translation tasks
Empirical Performance and GeneralizationThe Transformer model achieves state-of-the-art BLEU scores on both English-to-German (EN-DE) and English-to-French (EN-FR) translation tasks while requiring significantly lower training costs compared to previous architectures.
LessonF1 scores on English constituency parsing
Empirical Performance and GeneralizationThe Transformer generalizes well to English constituency parsing, achieving competitive F1 scores on Section 23 of WSJ under both discriminative (WSJ only) and semi-supervised training setups.
VisualF1 scores on English constituency parsing
Empirical Performance and GeneralizationExplore paper-reported results and reconciled deterministic comparisons.
VisualBLEU scores and training costs on WMT 2014 translation tasks
Empirical Performance and GeneralizationExplore paper-reported results and reconciled deterministic comparisons.
VisualF1 scores on English constituency parsing: F1
Empirical Performance and GeneralizationCompare reconciled F1 values reported by the paper.
VisualBLEU scores and training costs on WMT 2014 translation tasks: BLEU
Empirical Performance and GeneralizationCompare reconciled BLEU values reported by the paper.
QuizTest your understanding
Comprehensive Assessment12 questions grounded in this paper section.