Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Injecting Sequence Order: Positional Encodings

Sinusoidal Positional Encoding (Even Dimensions)

Source equation

$\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})$

In non-recurrent architectures like the Transformer, sequence order is not implicitly captured by the network structure. To inject positional information, sinusoidal positional encodings are added to the input embeddings. This equation defines the encoding value for even dimensions of the positional vector.

Sources

equation

PE(pos,2i)=sin(pos/100002i/dmodel)\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})

\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})

$\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})$

Sources

equation

PE(pos,2i)=sin(pos/100002i/dmodel)\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})

\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})

The positional encoding uses a combination of sine and cosine functions of different frequencies. For any given position $pos$ and even dimension index $2i$ , the value is computed using a sine function whose wavelength forms a geometric progression from $2\pi$ to $10000 \cdot 2\pi$ . This allows the model to easily learn to attend by relative positions, since for any fixed offset $k$ , $PE_{(pos+k)}$ can be represented as a linear function of $PE_{(pos)}$ .

Sources

equation

PE(pos,2i)=sin(pos/100002i/dmodel)\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})

\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})

Implementation detail

Illustrative Calculation

Let's calculate the positional encoding value for:

Position $pos = 1$
Dimension index $2i = 0$ (meaning $i = 0$ )
Model dimension $d_{\text{model}} = 512$

Step 1: Calculate the denominator $10000^{2i/d_{\text{model}}} = 10000^{0/512} = 10000^0 = 1$

Step 2: Calculate the argument of the sine function $\theta = \frac{pos}{10000^{2i/d_{\text{model}}}} = \frac{1}{1} = 1$

Step 3: Compute the sine value $PE_{(1, 0)} = \sin(1) \approx 0.84147$

Sources

equation

PE(pos,2i)=sin(pos/100002i/dmodel)\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})

\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})

$pos$: The position of the token in the sequence · scalar
$i$: The index of the dimension (where $2i$ represents even dimensions) · scalar
$d_{\text{model}}$: The total dimensionality of the model's hidden states · scalar
$PE_{(pos,2i)}$: The positional encoding value at position $pos$ for the even dimension $2i$ · scalar