Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Introduction to the Transformer Architecture

At a glance

Introduction to the Transformer Architecture

Sources

S3.F1

Figure 1: The Transformer - model architecture.

S3.p1.4

Most competitive neural sequence transduction models have an encoder-decoder structure [5, 2, 35]. Here, the encoder maps an input sequence of symbol representations (x1,…,xn)(x_{1},...,x_{n}) to a sequence of continuous representations 𝐳=(z1,…,zn)\mathbf{z}=(z_{1},...,z_{n}). Given 𝐳\mathbf{z}, the decoder then generates an output sequence (y1,…,ym)(y_{1},...,y_{m}) of symbols one element at a time. At each step the model is auto-regressive [10], consuming the previously generated symbols as additional input when generating the next.

S3.p2.1

The Transformer follows this overall architecture using stacked self-attention and point-wise, fully connected layers for both the encoder and decoder, shown in the left and right halves of Figure 1, respectively.

The Transformer model relies on an encoder-decoder structure. The encoder maps an input sequence of symbol representations $(x_1, \dots, x_n)$ to a sequence of continuous representations $\mathbf{z} = (z_1, \dots, z_n)$ . Given $\mathbf{z}$ , the decoder then generates an output sequence $(y_1, \dots, y_m)$ of symbols one element at a time in an auto-regressive manner.

Sources

S3.p1.4

Most competitive neural sequence transduction models have an encoder-decoder structure [5, 2, 35]. Here, the encoder maps an input sequence of symbol representations (x1,…,xn)(x_{1},...,x_{n}) to a sequence of continuous representations 𝐳=(z1,…,zn)\mathbf{z}=(z_{1},...,z_{n}). Given 𝐳\mathbf{z}, the decoder then generates an output sequence (y1,…,ym)(y_{1},...,y_{m}) of symbols one element at a time. At each step the model is auto-regressive [10], consuming the previously generated symbols as additional input when generating the next.

This architecture is built using stacked self-attention and point-wise, fully connected layers for both the encoder and decoder, as illustrated in the model architecture.

Sources

S3.F1

Figure 1: The Transformer - model architecture.

S3.p2.1

The Transformer follows this overall architecture using stacked self-attention and point-wise, fully connected layers for both the encoder and decoder, shown in the left and right halves of Figure 1, respectively.