Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Injecting Order with Positional Encoding

Sinusoidal Positional Encoding (Even)

Source equation

$\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})$

In sequence transduction models like the Transformer, which lack recurrence or convolution, positional encodings are added to the input embeddings to inject information about the order of tokens. This equation defines the sinusoidal positional encoding for even dimensions.

Sources

equation

PE(pos,2i)=sin(pos/100002i/dmodel)\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})

\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})

$\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})$

Sources

equation

PE(pos,2i)=sin(pos/100002i/dmodel)\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})

\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})

Deep dive

Walkthrough and Symbol Definitions

$PE_{(pos,2i)}$ : The positional encoding value at sequence position $pos$ and even dimension index $2i$ .
$pos$ : The position of the token in the sequence (0-indexed).
$i$ : The index of the frequency dimension, where $2i \in [0, d_{\text{model}})$ .
$d_{\text{model}}$ : The total dimensionality of the model's hidden states and embeddings.
$sin$ : The standard trigonometric sine function.

Sources

equation

PE(pos,2i)=sin(pos/100002i/dmodel)\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})

\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})

Implementation detail

Illustrative Example

Let's calculate the positional encoding value for:

$pos = 1$
$i = 0$ (which corresponds to the first even dimension, $2i = 0$ )
$d_{\text{model}} = 512$

Step-by-step calculation:

Calculate the denominator exponent: $\frac{2i}{d_{\text{model}}} = \frac{0}{512} = 0$ .
Calculate the base value: $10000^0 = 1$ .
Compute the inner term: $\frac{pos}{1} = \frac{1}{1} = 1$ .
Apply the sine function: $\sin(1) \approx 0.84147098$ .

Sources

equation

PE(pos,2i)=sin(pos/100002i/dmodel)\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})

\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})

$PE_{(pos,2i)}$: Positional encoding value at position pos and even dimension 2i · scalar
$pos$: The position of the token in the sequence · scalar
$i$: The index variable used to compute the dimension index 2i · scalar
$d_{\text{model}}$: The total dimensionality of the model embeddings · scalar