You are reading immutable version 28. The current guide may be newer.

Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Multi-Head Attention Mechanism

Multi-Head Attention Mechanism

Multi-Head Attention Mechanism

Multi-Head Attention Mechanism

Sources

S3.SS2.SSS2.p1.6

Instead of performing a single attention function with dmodeld_{\text{model}}-dimensional keys, values and queries, we found it beneficial to linearly project the queries, keys and values hh times with different, learned linear projections to dkd_{k}, dkd_{k} and dvd_{v} dimensions, respectively. On each of these projected versions of queries, keys and values we then perform the attention function in parallel, yielding dvd_{v}-dimensional output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2.

Instead of performing a single attention function with dmodeld_{\text{model}}-dimensional queries, keys, and values, the Transformer utilizes Multi-Head Attention. This mechanism linearly projects the queries, keys, and values hh times with different, learned linear projections to dkd_k, dkd_k, and dvd_v dimensions, respectively. This allows the model to jointly attend to information from different representation subspaces at different positions, whereas a single attention head would average these representations and inhibit such multi-faceted focus.

Sources

S3.SS2.SSS2.p1.6

Instead of performing a single attention function with dmodeld_{\text{model}}-dimensional keys, values and queries, we found it beneficial to linearly project the queries, keys and values hh times with different, learned linear projections to dkd_{k}, dkd_{k} and dvd_{v} dimensions, respectively. On each of these projected versions of queries, keys and values we then perform the attention function in parallel, yielding dvd_{v}-dimensional output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2.

S3.SS2.SSS2.p2.1

Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this.
Implementation detail

MultiHead(Q,K,V)=Concat(head1,...,headh)WO\displaystyle\mathrm{MultiHead}(Q,K,V)=\mathrm{Concat}(\mathrm{head_{1}},...,\mathrm{head_{h}})W^{O}

Sources

equation

MultiHead​(Q,K,V)\displaystyle\mathrm{MultiHead}(Q,K,V) =Concat​(head1,…,headh)​WO\displaystyle=\mathrm{Concat}(\mathrm{head_{1}},...,\mathrm{head_{h}})W^{O}
\displaystyle\mathrm{MultiHead}(Q,K,V)=\mathrm{Concat}(\mathrm{head_{1}},...,\mathrm{head_{h}})W^{O}

equation

where​headi\displaystyle\text{where}~\mathrm{head_{i}} =Attention​(Q​WiQ,K​WiK,V​WiV)\displaystyle=\mathrm{Attention}(QW^{Q}_{i},KW^{K}_{i},VW^{V}_{i})
\displaystyle\text{where}~\mathrm{head_{i}}=\mathrm{Attention}(QW^{Q}_{i},KW^{K}_{i},VW^{V}_{i})
Implementation detail

where each individual head is computed as:

Sources

equation

where​headi\displaystyle\text{where}~\mathrm{head_{i}} =Attention​(Q​WiQ,K​WiK,V​WiV)\displaystyle=\mathrm{Attention}(QW^{Q}_{i},KW^{K}_{i},VW^{V}_{i})
\displaystyle\text{where}~\mathrm{head_{i}}=\mathrm{Attention}(QW^{Q}_{i},KW^{K}_{i},VW^{V}_{i})
Implementation detail

where headi=Attention(QWiQ,KWiK,VWiV)\displaystyle\text{where}~\mathrm{head_{i}}=\mathrm{Attention}(QW^{Q}_{i},KW^{K}_{i},VW^{V}_{i})

Sources

equation

where​headi\displaystyle\text{where}~\mathrm{head_{i}} =Attention​(Q​WiQ,K​WiK,V​WiV)\displaystyle=\mathrm{Attention}(QW^{Q}_{i},KW^{K}_{i},VW^{V}_{i})
\displaystyle\text{where}~\mathrm{head_{i}}=\mathrm{Attention}(QW^{Q}_{i},KW^{K}_{i},VW^{V}_{i})