Study the paper

Deep Residual Learning for Image Recognition

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Mathematical Formulation of ResNets

A residual building block can be defined formally using shortcut connections. For a set of stacked layers that learn a residual mapping $\mathcal{F}(\mathbf{x}, \{W_i\})$ , the output $\mathbf{y}$ is computed by adding the input $\mathbf{x}$ directly to the output of the stacked layers:

Sources

S3.E1

𝐲=ℱ(𝐱,{Wi})+𝐱.𝐲ℱ𝐱subscript𝑊𝑖𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}. (1)

\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}

$\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}$

Sources

S3.E1

𝐲=ℱ(𝐱,{Wi})+𝐱.𝐲ℱ𝐱subscript𝑊𝑖𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}. (1)

\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}

This formulation introduces no extra parameters or computational complexity. However, when the dimensions of the input $\mathbf{x}$ and the residual output $\mathcal{F}$ differ (for example, when changing channel depths), a projection shortcut is used to match dimensions:

Sources

S3.E2

𝐲=ℱ(𝐱,{Wi})+Ws𝐱.𝐲ℱ𝐱subscript𝑊𝑖subscript𝑊𝑠𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)

\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}

$\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}$

Sources

S3.E2

𝐲=ℱ(𝐱,{Wi})+Ws𝐱.𝐲ℱ𝐱subscript𝑊𝑖subscript𝑊𝑠𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)

\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}

Here, $W_s$ is a projection matrix used solely to align dimensions. In practice, we can compare three options: (A) using zero-padding shortcuts for increasing dimensions (parameter-free), (B) using projection shortcuts only for increasing dimensions and identity shortcuts otherwise, or (C) using projection shortcuts for all connections.

Sources

S4.SS1.p9.1

Identity vs. Projection Shortcuts. We have shown that parameter-free, identity shortcuts help with training. Next we investigate projection shortcuts (Eqn.(2)). In Table 3 we compare three options: (A) zero-padding shortcuts are used for increasing dimensions, and all shortcuts are parameter-free (the same as Table 2 and Fig. 4 right); (B) projection shortcuts are used for increasing dimensions, and other shortcuts are identity; and (C) all shortcuts are projections.