Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Injecting Sequence Order via Positional Encodings

Sinusoidal Positional Encoding (Even Dimensions)

Source equation

$\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})$

Since the Transformer contains no recurrence or convolution, positional encodings are added to the input embeddings to inject information about the relative or absolute position of tokens in the sequence.

Sources

equation

PE(pos,2i)=sin(pos/100002i/dmodel)\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})

\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})

$\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})$

Sources

equation

PE(pos,2i)=sin(pos/100002i/dmodel)\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})

\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})

Implementation detail

Let's calculate a single positional encoding value deterministically:

Token position $pos = 1$
Dimension index $2i = 0$ (meaning $i = 0$ )
Model dimension $d_{\text{model}} = 512$

Compute the denominator: $10000^{2i/d_{\text{model}}} = 10000^{0/512} = 10000^0 = 1.0$
Compute the argument of the sine function: $\frac{pos}{10000^{2i/d_{\text{model}}}} = \frac{1}{1.0} = 1.0$
Compute the sine value: $PE_{(1, 0)} = \sin(1.0) \approx 0.84147098$

Sources

equation

PE(pos,2i)=sin(pos/100002i/dmodel)\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})

\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})

$pos$: Position of the token in the sequence · scalar
$i$: Index of the dimension · scalar
$d_{\text{model}}$: Dimensionality of the model embeddings · scalar
$PE_{(pos,2i)}$: Positional encoding value at position pos and even dimension 2i · scalar