Attention Is All You Need

Introduction to the Transformer Architecture

At a glance

Introduction to the Transformer Architecture

Sequential Computation Bottleneck in Recurrent Models

The Sequential Bottleneck of Recurrent Architectures

The Scaled Dot-Product Attention mechanism computes attention weights over a set of values based on queries and keys. The scaling factor sqrt d k is introduced to prevent the dot products from growing extremely large in magnitude for high-dimensional keys, which would otherwise push the softmax function into regions with extremely small gradients.

Illustrative teaching scenarios: attention scaling (not paper-reported)

Compare fixed illustrative teaching scenarios; these are not paper-reported results.

Illustrative teaching scenarios: attention concentration (not paper-reported)

Trace fixed illustrative teaching scenarios; these are not paper-reported results.

How attention redistributes weight

Illustrative attention weights computed from a fixed teaching example; these are not paper-reported values.

Explore attention score scaling

Move across fixed illustrative scenarios to see how score scaling changes concentration.

Scaled Dot-Product Attention: step by step

Follow the existing bounded walkthrough in its intended sequence.

Multi-Head Attention

Multi-Head Attention Mechanism

Multi-Head Attention

The Multi-Head Attention mechanism allows the model to jointly attend to information from different representation subspaces at different positions. Instead of performing a single attention function with d text model -dimensional queries, keys, and values, we project them h times with different, learned linear projections.

Multi-Head Attention: step by step

Multi-Head Attention

Follow the existing bounded walkthrough in its intended sequence.

Injecting Sequence Order: Positional Encodings

Sinusoidal Positional Encodings

Injecting Sequence Order: Positional Encodings

Sinusoidal Positional Encoding (Even Dimensions)

In non-recurrent architectures like the Transformer, sequence order is not implicitly captured by the network structure. To inject positional information, sinusoidal positional encodings are added to the input embeddings. This equation defines the encoding value for even dimensions of the positional vector.

Injecting Sequence Order: Positional Encodings

Sinusoidal Positional Encoding (Even Dimensions): step by step

Follow the existing bounded walkthrough in its intended sequence.

Empirical Results and Performance

Empirical Evaluation and Efficiency

Empirical Results and Performance

Machine Translation Performance and Training Costs

The Transformer model achieves state-of-the-art BLEU scores on both English-to-German (EN-DE) and English-to-French (EN-FR) translation tasks while requiring significantly lower training costs compared to previous recurrent or convolutional architectures.

Empirical Results and Performance

English Constituency Parsing Results

To evaluate if the Transformer generalizes well to other tasks, it was evaluated on English constituency parsing on Section 23 of WSJ. Despite the lack of task-specific tuning, the Transformer (4 layers) achieves strong results in both WSJ-only discriminative and semi-supervised settings.