Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Multi-Head Attention Mechanics

Mechanics of Multi-Head Attention

Sources

S3.SS2.SSS2.p1.6

Instead of performing a single attention function with dmodeld_{\text{model}}-dimensional keys, values and queries, we found it beneficial to linearly project the queries, keys and values hh times with different, learned linear projections to dkd_{k}, dkd_{k} and dvd_{v} dimensions, respectively. On each of these projected versions of queries, keys and values we then perform the attention function in parallel, yielding dvd_{v}-dimensional output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2.

Instead of performing a single attention function with $d_{\text{model}}$ -dimensional queries, keys, and values, Multi-Head Attention linearly projects them $h$ times using different learned linear projections to $d_k$ , $d_k$ , and $d_v$ dimensions. The attention function is then applied to these projected representations in parallel, yielding $d_v$ -dimensional outputs.

Sources

S3.SS2.SSS2.p1.6

Instead of performing a single attention function with dmodeld_{\text{model}}-dimensional keys, values and queries, we found it beneficial to linearly project the queries, keys and values hh times with different, learned linear projections to dkd_{k}, dkd_{k} and dvd_{v} dimensions, respectively. On each of these projected versions of queries, keys and values we then perform the attention function in parallel, yielding dvd_{v}-dimensional output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2.

Implementation detail

$\mathrm{MultiHead}(Q,K,V) = \mathrm{Concat}(\mathrm{head}_1, \dots, \mathrm{head}_h)W^O \quad \text{where } \mathrm{head}_i = \mathrm{Attention}(QW^Q_i, KW^K_i, VW^V_i)$

Sources

equation

MultiHead(Q,K,V)\displaystyle\mathrm{MultiHead}(Q,K,V) =Concat(head1,…,headh)WO\displaystyle=\mathrm{Concat}(\mathrm{head_{1}},...,\mathrm{head_{h}})W^{O}

\displaystyle\mathrm{MultiHead}(Q,K,V)=\mathrm{Concat}(\mathrm{head_{1}},...,\mathrm{head_{h}})W^{O}

equation

whereheadi\displaystyle\text{where}~\mathrm{head_{i}} =Attention(QWiQ,KWiK,VWiV)\displaystyle=\mathrm{Attention}(QW^{Q}_{i},KW^{K}_{i},VW^{V}_{i})

\displaystyle\text{where}~\mathrm{head_{i}}=\mathrm{Attention}(QW^{Q}_{i},KW^{K}_{i},VW^{V}_{i})

Deep dive

By projecting the inputs into multiple lower-dimensional subspaces, the model can jointly attend to information from different representation subspaces at different positions. A single attention head would instead average these signals, inhibiting the model's capacity to capture diverse dependencies.

Sources

S3.SS2.SSS2.p2.1

Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this.