Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Scaled Dot-Product Attention

Source equation

$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V$

The Scaled Dot-Product Attention mechanism computes attention weights by taking the dot product of queries with keys, scaling them by the square root of the key dimension, applying a softmax function, and using the resulting weights to compute a weighted sum of the values.

Sources

S3.E1

Attention(Q,K,V)=softmax(QKTdk)V\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V (1)

\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V

$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V$

Sources

S3.E1

Attention(Q,K,V)=softmax(QKTdk)V\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V (1)

\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V

Deep dive

To compute the attention matrix:

Compute the dot products of the queries $Q$ and keys $K$ by calculating $Q K^T$ .
Scale the dot products by dividing each element by $\sqrt{d_k}$ to prevent extremely large values that lead to vanishing gradients in the softmax function.
Apply the softmax function row-wise to obtain the attention weights.
Multiply the attention weights by the values $V$ to produce the final output matrix.

Sources

S3.E1

Attention(Q,K,V)=softmax(QKTdk)V\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V (1)

\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V

Implementation detail

Consider a simplified deterministic example with $d_k = 4$ (so $\sqrt{d_k} = 2$ ). Let the query matrix $Q$ and key matrix $K$ be:

$Q = \begin{pmatrix} 2 & 0 & 2 & 0 \end{pmatrix}, \quad K = \begin{pmatrix} 2 & 0 & 2 & 0 \end{pmatrix}$

Then, the dot product is:

$Q K^T = \begin{pmatrix} 2 \cdot 2 + 0 \cdot 0 + 2 \cdot 2 + 0 \cdot 0 \end{pmatrix} = \begin{pmatrix} 8 \end{pmatrix}$

Scaling by $\sqrt{d_k} = 2$ yields:

$\frac{Q K^T}{\sqrt{d_k}} = \frac{8}{2} = 4$

Applying the softmax (which is trivial for a single element, yielding $1.0$ ) and multiplying by a value vector $V = \begin{pmatrix} 5 \end{pmatrix}$ results in:

$\mathrm{Attention}(Q,K,V) = 1.0 \cdot 5 = 5$

Sources

S3.E1

Attention(Q,K,V)=softmax(QKTdk)V\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V (1)

\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V

$Q$: Query matrix of shape (d_seq, d_k) · [d_seq, d_k]
$K$: Key matrix of shape (d_seq, d_k) · [d_seq, d_k]
$V$: Value matrix of shape (d_seq, d_v) · [d_seq, d_v]
$d_k$: Dimension of the keys · scalar
$softmax$: Softmax activation function applied row-wise · function