Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Scaled Dot-Product Attention

Mathematical Formulation

Sources

S3.E1

Attention(Q,K,V)=softmax(QKTdk)V\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V (1)

\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V

The Scaled Dot-Product Attention mechanism maps a set of queries, keys, and values to an output. The queries, keys, and values are packed into matrices $Q$ , $K$ , and $V$ respectively. The attention weights are computed using the dot product of the queries and keys, scaled by the square root of the key dimension $d_k$ , and normalized with a softmax function.

Sources

S3.E1

Attention(Q,K,V)=softmax(QKTdk)V\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V (1)

\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V

$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V$

Sources

S3.E1

Attention(Q,K,V)=softmax(QKTdk)V\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V (1)

\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V

The Necessity of the Scaling Factor

Sources

S3.SS2.SSS1.p5.4

While for small values of dkd_{k} the two mechanisms perform similarly, additive attention outperforms dot product attention without scaling for larger values of dkd_{k} [3]. We suspect that for large values of dkd_{k}, the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients 111To illustrate why the dot products get large, assume that the components of qq and kk are independent random variables with mean 0 and variance 11. Then their dot product, q⋅k=∑i=1dkqikiq\cdot k=\sum_{i=1}^{d_{k}}q_{i}k_{i}, has mean 0 and variance dkd_{k}.. To counteract this effect, we scale the dot products by 1dk\frac{1}{\sqrt{d_{k}}}.

For small values of $d_k$ , additive attention and dot-product attention perform similarly. However, for larger values of $d_k$ , dot-product attention without scaling is outperformed by additive attention. This occurs because the dot products grow large in magnitude, pushing the softmax function into regions with extremely small gradients.

Sources

S3.SS2.SSS1.p5.4

While for small values of dkd_{k} the two mechanisms perform similarly, additive attention outperforms dot product attention without scaling for larger values of dkd_{k} [3]. We suspect that for large values of dkd_{k}, the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients 111To illustrate why the dot products get large, assume that the components of qq and kk are independent random variables with mean 0 and variance 11. Then their dot product, q⋅k=∑i=1dkqikiq\cdot k=\sum_{i=1}^{d_{k}}q_{i}k_{i}, has mean 0 and variance dkd_{k}.. To counteract this effect, we scale the dot products by 1dk\frac{1}{\sqrt{d_{k}}}.

Variance Analysis

To understand why dot products grow large, assume that the components of a query vector $q$ and a key vector $k$ are independent random variables with mean $0$ and variance $1$ . Their dot product, $q \cdot k = \sum_{i=1}^{d_k} q_i k_i$ , has a mean of $0$ and a variance of $d_k$ . Scaling the dot product by $1/\sqrt{d_k}$ pulls the variance back to $1$ , preventing the softmax function from saturating.

Sources

S3.SS2.SSS1.p5.4

While for small values of dkd_{k} the two mechanisms perform similarly, additive attention outperforms dot product attention without scaling for larger values of dkd_{k} [3]. We suspect that for large values of dkd_{k}, the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients 111To illustrate why the dot products get large, assume that the components of qq and kk are independent random variables with mean 0 and variance 11. Then their dot product, q⋅k=∑i=1dkqikiq\cdot k=\sum_{i=1}^{d_{k}}q_{i}k_{i}, has mean 0 and variance dkd_{k}.. To counteract this effect, we scale the dot products by 1dk\frac{1}{\sqrt{d_{k}}}.