You are reading immutable version 39. The current guide may be newer.

Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Injecting Sequence Order: Positional Encodings

Sinusoidal Positional Encoding (Even Dimensions): step by step

Interactive study view

Sinusoidal Positional Encoding (Even Dimensions): step by step

Illustrative teaching data

Follow the existing bounded walkthrough in its intended sequence.

Step 1 of 5

  1. Step 1

    Identify inputs: pos = 1, i = 0, d_model = 512.

  2. Step 2

    Compute the exponent: 2 * i / d_model = 0 / 512 = 0.

  3. Step 3

    Compute the denominator: 10000^0 = 1.

  4. Step 4

    Compute the division: pos / 1 = 1 / 1 = 1.

  5. Step 5

    Compute the sine of 1 radian: sin(1) ≈ 0.8414709848.