You are reading immutable version 39. The current guide may be newer.

Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Multi-Head Attention

Multi-Head Attention: step by step

Interactive study view

Multi-Head Attention: step by step

Illustrative teaching data

Follow the existing bounded walkthrough in its intended sequence.

Step 1 of 4

  1. Step 1

    Project Q, K, V into h subspaces using learned projection matrices.

  2. Step 2

    Apply Scaled Dot-Product Attention to each projected subspace independently to obtain head_1 through head_h.

  3. Step 3

    Concatenate the outputs of all heads along the feature dimension.

  4. Step 4

    Multiply the concatenated matrix by the output projection matrix W^O to obtain the final representation.