You are reading immutable version 5. The current guide may be newer.

Study the paper

Deep Residual Learning for Image Recognition

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Mathematics of Identity Shortcuts and Dimension Matching

Residual Block with Identity Shortcut

Residual Block with Identity Shortcut

Source equation

y=F(x,{Wi})+x\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}

This equation defines the fundamental formulation of a residual learning block. Instead of directly fitting an underlying mapping with a stack of layers, the network is designed to approximate a residual mapping F(x,{Wi})=y−x\mathcal{F}(\mathbf{x}, \{W_i\}) = \mathbf{y} - \mathbf{x}. The original mapping is recast into F(x)+x\mathcal{F}(\mathbf{x}) + \mathbf{x}, which is realized by a feedforward neural network with shortcut connections.

Sources

S3.E1

𝐲=ℱ​(𝐱,{Wi})+𝐱.𝐲ℱ𝐱subscript𝑊𝑖𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}. (1)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}

y=F(x,{Wi})+x\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}

Sources

S3.E1

𝐲=ℱ​(𝐱,{Wi})+𝐱.𝐲ℱ𝐱subscript𝑊𝑖𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}. (1)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}

The formulation adds the input vector x\mathbf{x} directly to the output of the residual function F\mathcal{F}. This identity shortcut connection introduces neither extra parameters nor computational complexity, allowing the network to be trained end-to-end via backpropagation.

Sources

S3.E1

𝐲=ℱ​(𝐱,{Wi})+𝐱.𝐲ℱ𝐱subscript𝑊𝑖𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}. (1)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}
Implementation detail

Illustrative Numerical Example

Let us compute a forward pass for a single-channel 1D feature vector:

  • Input vector: x=[1.02.0]\mathbf{x} = \begin{bmatrix} 1.0 \\ 2.0 \end{bmatrix}
  • Residual function: F(x,{Wi})=[−0.50.3]\mathcal{F}(\mathbf{x}, \{W_i\}) = \begin{bmatrix} -0.5 \\ 0.3 \end{bmatrix}

Applying the residual formula: y=F(x,{Wi})+x=[−0.50.3]+[1.02.0]=[0.52.3]\mathbf{y} = \mathcal{F}(\mathbf{x}, \{W_i\}) + \mathbf{x} = \begin{bmatrix} -0.5 \\ 0.3 \end{bmatrix} + \begin{bmatrix} 1.0 \\ 2.0 \end{bmatrix} = \begin{bmatrix} 0.5 \\ 2.3 \end{bmatrix}

Sources

S3.E1

𝐲=ℱ​(𝐱,{Wi})+𝐱.𝐲ℱ𝐱subscript𝑊𝑖𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}. (1)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}
y\mathbf{y}
Output vector of the residual block · D-dimensional vector
x\mathbf{x}
Input vector to the residual block · D-dimensional vector
F\mathcal{F}
Residual mapping function to be learned · Function mapping from D-dimensional space to D-dimensional space
{Wi}\{W_{i}\}
Set of weights associated with the layers in the residual block · Set of weight matrices