Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

13 activities

Lesson

At a glance

Introduction to the Transformer Architecture

Lesson

The Encoder-Decoder Structure of the Transformer

Introduction to the Transformer Architecture

Traditional sequence transduction models rely on an encoder-decoder structure. The encoder maps an input sequence of symbol representations (x 1, u, x n) to a sequence of continuous representations mathbf z = (z 1, u, z n) . Given mathbf z , the decoder then generates an output sequence (y 1, u, y m) of symbols one element at a time, operating in an auto-regressive manner by consuming previously generated symbols as additional input.

Lesson

Scaled Dot-Product Attention

Scaled Dot-Product Attention Formulation

Lesson

Scaled Dot-Product Attention

Scaled Dot-Product Attention computes attention weights by taking the dot product of queries with keys, scaling them by the square root of the key dimension, applying a softmax function, and using the resulting weights to compute a weighted sum of values.

Lesson

Multi-Head Attention Mechanism

Instead of performing a single attention function with d text model -dimensional queries, keys, and values, Multi-Head Attention projects them h times using different, learned linear projections to d k , d k , and d v dimensions, respectively. The attention function is then performed in parallel across these projected versions, yielding d v -dimensional output values.

Lesson

Injecting Sequence Order via Positional Encodings

Because the Transformer contains no recurrence and no convolution, it lacks an inherent sense of sequence order. To make use of the order of the sequence, we must inject information about the relative or absolute position of the tokens. This is achieved by adding "positional encodings" of dimension d text model to the input embeddings at the bottoms of the encoder and decoder stacks.

Lesson

Sinusoidal Positional Encoding (Even Dimensions)

Injecting Sequence Order via Positional Encodings

Since the Transformer contains no recurrence or convolution, positional encodings are added to the input embeddings to inject information about the relative or absolute position of tokens in the sequence.

Lesson

Analyzing Computational Complexity and Path Lengths

Comparing Self-Attention, Recurrent, and Convolutional Layers

Self-attention layers connect all positions with a constant number of sequentially executed operations ( O(1) ), whereas recurrent layers require O(n) sequential operations. In terms of computational complexity, self-attention layers are faster than recurrent layers when the sequence length n is smaller than the representation dimensionality d , which is typical for state-of-the-art machine translation models.

Lesson

Limitations of Recurrent Neural Networks in Sequence Modeling

Comparing Self-Attention, Recurrent, and Convolutional Layers

In sequence modeling, recurrent neural networks process tokens sequentially, which creates a fundamental computational bottleneck. Because the computation of hidden state h t depends on h t-1 , operations cannot be easily parallelized across the sequence length n . This results in O(n) sequential operations, as opposed to O(1) for self-attention layers.

Lesson

Empirical Performance on Translation and Parsing

Empirical Performance and Generalization

On the WMT 2014 English-to-German translation task, the big Transformer model achieves a state-of-the-art BLEU score of 28.4, outperforming previous models and ensembles by more than 2.0 BLEU. The base model also outperforms previous models at a fraction of the training cost (requiring only 3.3 cdot 10^ 18 FLOPs compared to over 10^ 19 or 10^ 20 FLOPs for competitive architectures).

Lesson

Table 2: BLEU scores and training costs for translation tasks

Empirical Performance and Generalization

The Transformer model achieves state-of-the-art BLEU scores on the WMT 2014 English-to-German (EN-DE) and English-to-French (EN-FR) translation tasks while requiring significantly lower training costs compared to previous recurrent or convolutional architectures.

Lesson

Table 4: English constituency parsing results

Empirical Performance and Generalization

The Transformer generalizes well to English constituency parsing, achieving strong performance on Section 23 of WSJ in both discriminative (WSJ only) and semi-supervised settings.

Quiz

Test your understanding

Transformer Architecture Assessment

12 questions grounded in this paper section.