You are reading immutable version 39. The current guide may be newer.

Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Scaled Dot-Product Attention

Scaled Dot-Product Attention: step by step

Interactive study view

Scaled Dot-Product Attention: step by step

Illustrative teaching data

Follow the existing bounded walkthrough in its intended sequence.

Step 1 of 4

  1. Step 1

    Compute dot product of Q and K^T to get raw scores [2.0, 0.0].

  2. Step 2

    Scale raw scores by dividing by sqrt(d_k) = 2.0 to get [1.0, 0.0].

  3. Step 3

    Apply softmax to obtain attention weights [0.731, 0.269].

  4. Step 4

    Compute weighted sum of values V to obtain the final output [12.69].