Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Multi-Head Attention Mechanism

Instead of performing a single attention function with $d_{\text{model}}$ -dimensional queries, keys, and values, Multi-Head Attention projects them $h$ times using different, learned linear projections to $d_k$ , $d_k$ , and $d_v$ dimensions, respectively. The attention function is then performed in parallel across these projected versions, yielding $d_v$ -dimensional output values.

Sources

S3.SS2.SSS2.p1.6

Instead of performing a single attention function with dmodeld_{\text{model}}-dimensional keys, values and queries, we found it beneficial to linearly project the queries, keys and values hh times with different, learned linear projections to dkd_{k}, dkd_{k} and dvd_{v} dimensions, respectively. On each of these projected versions of queries, keys and values we then perform the attention function in parallel, yielding dvd_{v}-dimensional output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2.

$\mathrm{MultiHead}(Q,K,V) = \mathrm{Concat}(\mathrm{head_1}, \dots, \mathrm{head_h})W^O \text{ where } \mathrm{head_i} = \mathrm{Attention}(QW^Q_i, KW^K_i, VW^V_i)$

Sources

equation

MultiHead(Q,K,V)\displaystyle\mathrm{MultiHead}(Q,K,V) =Concat(head1,…,headh)WO\displaystyle=\mathrm{Concat}(\mathrm{head_{1}},...,\mathrm{head_{h}})W^{O}

\displaystyle\mathrm{MultiHead}(Q,K,V)

equation

whereheadi\displaystyle\text{where}~\mathrm{head_{i}} =Attention(QWiQ,KWiK,VWiV)\displaystyle=\mathrm{Attention}(QW^{Q}_{i},KW^{K}_{i},VW^{V}_{i})

\displaystyle\text{where}~\mathrm{head_{i}}

Deep dive

Why Multi-Head Attention?

Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. A single attention head would average these representations, inhibiting the model's ability to capture diverse positional relationships.

Sources

S3.SS2.SSS2.p2.1

Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this.