You are reading immutable version 40. The current guide may be newer.

Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Multi-Head Attention Mechanics

Multi-Head Attention: step by step

Interactive study view

Multi-Head Attention: step by step

Illustrative teaching data

Follow the existing bounded walkthrough in its intended sequence.

Step 1 of 3

  1. Step 1

    Concatenate head_1 and head_2 to get [1.0, 2.0, 3.0, 4.0]

  2. Step 2

    Multiply by the projection matrix W^O to obtain the final output vector [4.0, 6.0]

  3. Step 3

    Sum the elements of the output vector to produce a single scalar result: 4.0 + 6.0 = 10.0