Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Scaled dot-product attention

Source equation

$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V$

Sources

S3.E1

Attention(Q,K,V)=softmax(QKTdk)V\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V (1)

\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V

Deep dive

The product $QK^\top$ forms query-key scores, division by $\sqrt{d_k}$ scales them, softmax normalizes each score row, and multiplication by $V$ returns weighted values.

Sources

S3.E1

Attention(Q,K,V)=softmax(QKTdk)V\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V (1)

\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V

Implementation detail

Illustrative scalar case: if the unscaled score is 2 and $d_k=4$ , the scaled score is 1 before softmax.

$Q$: Query matrix · n_q x d_k
$K$: Key matrix · n_k x d_k
$V$: Value matrix · n_k x d_v
$d_k$: Key-vector dimension