You are reading immutable version 4. The current guide may be newer.

Study the paper

Deep Residual Learning for Image Recognition

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Mathematical Formulation of Residual Learning

Residual Building Block with Identity Mapping

Residual Building Block with Identity Mapping

Source equation

y=F(x,{Wi})+x\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}

This equation defines the fundamental building block of Deep Residual Learning (ResNet). Instead of forcing stacked layers to directly fit a desired underlying mapping H(x)\mathcal{H}(\mathbf{x}), we let these layers approximate a residual mapping F(x,{Wi})=H(x)−x\mathcal{F}(\mathbf{x}, \{W_i\}) = \mathcal{H}(\mathbf{x}) - \mathbf{x}. The original mapping is recast into F(x)+x\mathcal{F}(\mathbf{x}) + \mathbf{x} via an identity shortcut connection.

Sources

S3.E1

𝐲=ℱ​(𝐱,{Wi})+𝐱.𝐲ℱ𝐱subscript𝑊𝑖𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}. (1)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}

y=F(x,{Wi})+x\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}

Sources

S3.E1

𝐲=ℱ​(𝐱,{Wi})+𝐱.𝐲ℱ𝐱subscript𝑊𝑖𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}. (1)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}

equation

𝐲=ℱ​(𝐱,{Wi})+𝐱.𝐲ℱ𝐱subscript𝑊𝑖𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}. (1)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}

The identity shortcut connection adds back the input x\mathbf{x} to the output of the residual function F\mathcal{F}. This operation requires no extra parameters and introduces no computational complexity beyond element-wise addition.

Sources

equation

𝐲=ℱ​(𝐱,{Wi})+𝐱.𝐲ℱ𝐱subscript𝑊𝑖𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}. (1)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}
Implementation detail

Illustrative Example

Let us compute a simple forward pass for a single-channel 2D vector input:

  • Input vector: x=[1.02.0]\mathbf{x} = \begin{bmatrix} 1.0 \\ 2.0 \end{bmatrix}
  • Residual function: F(x,{Wi})=W2⋅ReLU(W1x)\mathcal{F}(\mathbf{x}, \{W_i\}) = W_2 \cdot \text{ReLU}(W_1 \mathbf{x})
  • Weights: W1=[0.5−0.50.01.0],W2=[1.00.00.50.5]W_1 = \begin{bmatrix} 0.5 & -0.5 \\ 0.0 & 1.0 \end{bmatrix}, \quad W_2 = \begin{bmatrix} 1.0 & 0.0 \\ 0.5 & 0.5 \end{bmatrix}

Step 1: Compute W1xW_1 \mathbf{x}: W1x=[0.5(1.0)−0.5(2.0)0.0(1.0)+1.0(2.0)]=[−0.52.0]W_1 \mathbf{x} = \begin{bmatrix} 0.5(1.0) - 0.5(2.0) \\ 0.0(1.0) + 1.0(2.0) \end{bmatrix} = \begin{bmatrix} -0.5 \\ 2.0 \end{bmatrix}

Step 2: Apply ReLU activation: ReLU([−0.52.0])=[0.02.0]\text{ReLU}\left(\begin{bmatrix} -0.5 \\ 2.0 \end{bmatrix}\right) = \begin{bmatrix} 0.0 \\ 2.0 \end{bmatrix}

Step 3: Compute residual mapping F\mathcal{F}: F(x,{Wi})=W2[0.02.0]=[1.0(0.0)+0.0(2.0)0.5(0.0)+0.5(2.0)]=[0.01.0]\mathcal{F}(\mathbf{x}, \{W_i\}) = W_2 \begin{bmatrix} 0.0 \\ 2.0 \end{bmatrix} = \begin{bmatrix} 1.0(0.0) + 0.0(2.0) \\ 0.5(0.0) + 0.5(2.0) \end{bmatrix} = \begin{bmatrix} 0.0 \\ 1.0 \end{bmatrix}

Step 4: Add the identity shortcut connection x\mathbf{x}: y=F(x,{Wi})+x=[0.01.0]+[1.02.0]=[1.03.0]\mathbf{y} = \mathcal{F}(\mathbf{x}, \{W_i\}) + \mathbf{x} = \begin{bmatrix} 0.0 \\ 1.0 \end{bmatrix} + \begin{bmatrix} 1.0 \\ 2.0 \end{bmatrix} = \begin{bmatrix} 1.0 \\ 3.0 \end{bmatrix}

Sources

S3.E1

𝐲=ℱ​(𝐱,{Wi})+𝐱.𝐲ℱ𝐱subscript𝑊𝑖𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}. (1)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}
y\mathbf{y}
Output vector of the residual block · [D]
x\mathbf{x}
Input vector to the residual block · [D]
F\mathcal{F}
Residual mapping function to be learned · [D] -> [D]
WiW_i
Set of weight matrices associated with the layers in the block · List of [D_out, D_in]