You are reading immutable version 39. The current guide may be newer.

Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Multi-Head Attention

Multi-Head Attention

Multi-Head Attention

Multi-Head Attention Mechanism

Sources

S3.SS2.SSS2.p1.6

Instead of performing a single attention function with dmodeld_{\text{model}}-dimensional keys, values and queries, we found it beneficial to linearly project the queries, keys and values hh times with different, learned linear projections to dkd_{k}, dkd_{k} and dvd_{v} dimensions, respectively. On each of these projected versions of queries, keys and values we then perform the attention function in parallel, yielding dvd_{v}-dimensional output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2.

Instead of performing a single attention function with dmodeld_{\text{model}}-dimensional queries, keys, and values, Multi-Head Attention projects them into multiple lower-dimensional subspaces. This allows the model to jointly attend to information from different representation subspaces at different positions, which would otherwise be averaged out by a single attention head.

Sources

S3.SS2.SSS2.p2.1

Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this.
Implementation detail

MultiHead(Q,K,V)=Concat(head1,...,headh)WO\displaystyle\mathrm{MultiHead}(Q,K,V)=\mathrm{Concat}(\mathrm{head_{1}},...,\mathrm{head_{h}})W^{O}

Sources

equation

MultiHead​(Q,K,V)\displaystyle\mathrm{MultiHead}(Q,K,V) =Concat​(head1,…,headh)​WO\displaystyle=\mathrm{Concat}(\mathrm{head_{1}},...,\mathrm{head_{h}})W^{O}
\displaystyle\mathrm{MultiHead}(Q,K,V)=\mathrm{Concat}(\mathrm{head_{1}},...,\mathrm{head_{h}})W^{O}

equation

where​headi\displaystyle\text{where}~\mathrm{head_{i}} =Attention​(Q​WiQ,K​WiK,V​WiV)\displaystyle=\mathrm{Attention}(QW^{Q}_{i},KW^{K}_{i},VW^{V}_{i})
\displaystyle\text{where}~\mathrm{head_{i}}=\mathrm{Attention}(QW^{Q}_{i},KW^{K}_{i},VW^{V}_{i})
Implementation detail

where each head is computed as: headi=Attention(QWiQ,KWiK,VWiV)\mathrm{head}_i = \mathrm{Attention}(QW^Q_i, KW^K_i, VW^V_i) Here, the projections are learned parameter matrices WiQ∈Rdmodel×dkW^Q_i \in \mathbb{R}^{d_{\text{model}} \times d_k}, WiK∈Rdmodel×dkW^K_i \in \mathbb{R}^{d_{\text{model}} \times d_k}, WiV∈Rdmodel×dvW^V_i \in \mathbb{R}^{d_{\text{model}} \times d_v}, and WO∈Rhdv×dmodelW^O \in \mathbb{R}^{h d_v \times d_{\text{model}}}.

Sources

equation

where​headi\displaystyle\text{where}~\mathrm{head_{i}} =Attention​(Q​WiQ,K​WiK,V​WiV)\displaystyle=\mathrm{Attention}(QW^{Q}_{i},KW^{K}_{i},VW^{V}_{i})
\displaystyle\text{where}~\mathrm{head_{i}}=\mathrm{Attention}(QW^{Q}_{i},KW^{K}_{i},VW^{V}_{i})