Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Introduction to the Transformer Architecture

The Encoder-Decoder Structure of the Transformer

Traditional sequence transduction models rely on an encoder-decoder structure. The encoder maps an input sequence of symbol representations $(x_1, u, x_n)$ to a sequence of continuous representations $\mathbf{z} = (z_1, u, z_n)$ . Given $\mathbf{z}$ , the decoder then generates an output sequence $(y_1, u, y_m)$ of symbols one element at a time, operating in an auto-regressive manner by consuming previously generated symbols as additional input.

Sources

S3.p1.4

Most competitive neural sequence transduction models have an encoder-decoder structure [5, 2, 35]. Here, the encoder maps an input sequence of symbol representations (x1,…,xn)(x_{1},...,x_{n}) to a sequence of continuous representations 𝐳=(z1,…,zn)\mathbf{z}=(z_{1},...,z_{n}). Given 𝐳\mathbf{z}, the decoder then generates an output sequence (y1,…,ym)(y_{1},...,y_{m}) of symbols one element at a time. At each step the model is auto-regressive [10], consuming the previously generated symbols as additional input when generating the next.

The Transformer architecture follows this overall encoder-decoder structure but dispenses with recurrence and convolutions entirely. Instead, it relies on stacked self-attention and point-wise, fully connected layers for both the encoder (left half of the architecture) and decoder (right half of the architecture).

Sources

S3.p2.1

The Transformer follows this overall architecture using stacked self-attention and point-wise, fully connected layers for both the encoder and decoder, shown in the left and right halves of Figure 1, respectively.

S3.F1

Figure 1: The Transformer - model architecture.