You are reading immutable version 28. The current guide may be newer.

Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Multi-Head Attention Mechanism

Multi-Head Attention: step by step

Interactive study view

Multi-Head Attention: step by step

Illustrative teaching data

Follow the existing bounded walkthrough in its intended sequence.

Step 1 of 2

  1. Step 1

    Concatenate individual head outputs along the feature dimension.

  2. Step 2

    Multiply the concatenated representation by the output projection matrix W^O.