You are reading immutable version 10. The current guide may be newer.

Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Scaled Dot-Product Attention

Scaled Dot-Product Attention

Scaled Dot-Product Attention

Scaled Dot-Product Attention Formulation

Sources

S3.E1

Attention​(Q,K,V)=softmax​(Q​KTdk)​V\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V (1)
\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V

equation

Attention​(Q,K,V)=softmax​(Q​KTdk)​V\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V (1)
\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V

The attention mechanism maps queries, keys, and values to an output. In Scaled Dot-Product Attention, the input consists of queries and keys of dimension dkd_k, and values of dimension dvd_v. We compute the dot products of the query with all keys, divide each by dk\sqrt{d_k}, and apply a softmax function to obtain the weights on the values.

Sources

S3.E1

Attention​(Q,K,V)=softmax​(Q​KTdk)​V\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V (1)
\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V

S3.SS2.SSS1.p5.4

While for small values of dkd_{k} the two mechanisms perform similarly, additive attention outperforms dot product attention without scaling for larger values of dkd_{k} [3]. We suspect that for large values of dkd_{k}, the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients 111To illustrate why the dot products get large, assume that the components of qq and kk are independent random variables with mean 0 and variance 11. Then their dot product, q⋅k=∑i=1dkqi​kiq\cdot k=\sum_{i=1}^{d_{k}}q_{i}k_{i}, has mean 0 and variance dkd_{k}.. To counteract this effect, we scale the dot products by 1dk\frac{1}{\sqrt{d_{k}}}.

Attention(Q,K,V)=softmax(QKTdk)V\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V

Sources

S3.E1

Attention​(Q,K,V)=softmax​(Q​KTdk)​V\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V (1)
\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V

equation

Attention​(Q,K,V)=softmax​(Q​KTdk)​V\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V (1)
\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V

Why Scale by 1/dk1/\sqrt{d_k}?

For large values of dkd_k, the dot products grow large in magnitude. This pushes the softmax function into regions with extremely small gradients. Assuming components of qq and kk are independent random variables with mean 0 and variance 1, their dot product q⋅k=∑i=1dkqikiq \cdot k = \sum_{i=1}^{d_k} q_i k_i has mean 0 and variance dkd_k. Dividing by dk\sqrt{d_k} scales the variance back to 1 and counteracts this vanishing gradient effect.

Sources

S3.SS2.SSS1.p5.4

While for small values of dkd_{k} the two mechanisms perform similarly, additive attention outperforms dot product attention without scaling for larger values of dkd_{k} [3]. We suspect that for large values of dkd_{k}, the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients 111To illustrate why the dot products get large, assume that the components of qq and kk are independent random variables with mean 0 and variance 11. Then their dot product, q⋅k=∑i=1dkqi​kiq\cdot k=\sum_{i=1}^{d_{k}}q_{i}k_{i}, has mean 0 and variance dkd_{k}.. To counteract this effect, we scale the dot products by 1dk\frac{1}{\sqrt{d_{k}}}.