Attention Is All You Need

Introduction to the Transformer Architecture

At a glance

Introduction to the Transformer Architecture

Motivation for the Transformer

Recurrent neural networks (RNNs) process sequences sequentially, computing a hidden state h t as a function of the previous hidden state h t-1 and the current input. This step-by-step alignment prevents parallelization within a single training example, leading to severe computational bottlenecks for long sequences.

Introduction to the Transformer Architecture

2 cards for focused recall and explanation.

Scaled Dot-Product Attention

Mathematical Formulation

Scaled Dot-Product Attention

The Scaled Dot-Product Attention mechanism maps a set of query vectors, key vectors, and value vectors to an output. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key.

Scaled Dot-Product Attention

2 cards for focused recall and explanation.

Illustrative teaching scenarios: attention scaling (not paper-reported)

Compare fixed illustrative teaching scenarios; these are not paper-reported results.

Illustrative teaching scenarios: attention concentration (not paper-reported)

Trace fixed illustrative teaching scenarios; these are not paper-reported results.

How attention redistributes weight

Illustrative attention weights computed from a fixed teaching example; these are not paper-reported values.

Explore attention score scaling

Move across fixed illustrative scenarios to see how score scaling changes concentration.

Scaled Dot-Product Attention: step by step

Follow the existing bounded walkthrough in its intended sequence.

Multi-Head Attention Mechanics

Mechanics of Multi-Head Attention

Multi-Head Attention Mechanics

Multi-Head Attention

The Multi-Head Attention mechanism projects queries, keys, and values into multiple lower-dimensional subspaces, performs attention on each subspace in parallel, and then concatenates and projects the results back to the original dimension.

Multi-Head Attention Mechanics

2 cards for focused recall and explanation.

Multi-Head Attention Mechanics

Multi-Head Attention: step by step

Follow the existing bounded walkthrough in its intended sequence.

Injecting Sequence Order via Positional Encodings

Sinusoidal Positional Encodings

Because the Transformer contains no recurrence and no convolution, it is entirely permutation-invariant. To make use of the order of the sequence, we must inject information about the relative or absolute position of the tokens. This is achieved by adding "positional encodings" of dimension d text model directly to the input embeddings.

Injecting Sequence Order via Positional Encodings

2 cards for focused recall and explanation.

Comparing Self-Attention, Recurrent, and Convolutional Layers

Computational Complexity of Layer Types

To understand the computational advantages of self-attention, we compare its per-layer complexity, sequential operations, and maximum path length against recurrent and convolutional layers. Let n be the sequence length and d be the representation dimension.

Comparing Self-Attention, Recurrent, and Convolutional Layers

Restricted Self-Attention for Long Sequences

For extremely long sequences, self-attention can be restricted to a local neighborhood of size r centered around each output position. This restricted self-attention reduces complexity per layer to O(r cdot n cdot d) while increasing the maximum path length to O(n/r) .

Comparing Self-Attention, Recurrent, and Convolutional Layers

3 cards for focused recall and explanation.

Empirical Evaluation and Generalization

Empirical Performance and Generalization

The Transformer achieves state-of-the-art performance on translation tasks with significantly lower training costs. On the WMT 2014 English-to-German task, the big Transformer model achieves a BLEU score of 28.4, outperforming previous ensemble models while requiring only 2.3 cdot 10^ 19 FLOPs to train.

Empirical Evaluation and Generalization

Table 2: BLEU scores and training costs for translation tasks

The Transformer model achieves state-of-the-art results on translation tasks. On the English-to-German (EN-DE) translation task, the Transformer (big) model achieves a BLEU score of 28.4, while the base model achieves 27.3. On the English-to-French (EN-FR) task, the Transformer (big) model achieves a BLEU score of 41.8, and the base model achieves 38.1.

Empirical Evaluation and Generalization

Table 4: F1 scores for English constituency parsing

The Transformer generalizes well to English constituency parsing. When trained on WSJ only (discriminative setting), the 4-layer Transformer achieves an F1 score of 91.3. In the semi-supervised setting, the 4-layer Transformer achieves an F1 score of 92.7.