You are reading immutable version 3. The current guide may be newer.

Study the paper

Deep Residual Learning for Image Recognition

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Mathematical Formulation of ResNets

Identity Shortcut Connection

Identity Shortcut Connection

Source equation

y=F(x,{Wi})+x\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}

The fundamental formulation of deep residual learning introduces an identity shortcut connection. Instead of forcing stacked layers to directly fit a desired underlying mapping H(x)\mathcal{H}(\mathbf{x}), we let these layers approximate a residual mapping F(x,{Wi})=H(x)−x\mathcal{F}(\mathbf{x}, \{W_i\}) = \mathcal{H}(\mathbf{x}) - \mathbf{x}. The original mapping is recast into F(x,{Wi})+x\mathcal{F}(\mathbf{x}, \{W_i\}) + \mathbf{x}.

Sources

S3.E1

𝐲=ℱ​(𝐱,{Wi})+𝐱.𝐲ℱ𝐱subscript𝑊𝑖𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}. (1)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}

y=F(x,{Wi})+x\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}

Sources

S3.E1

𝐲=ℱ​(𝐱,{Wi})+𝐱.𝐲ℱ𝐱subscript𝑊𝑖𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}. (1)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}
Deep dive

This formulation allows the network to pass the input x\mathbf{x} directly through the shortcut connection, requiring the weight layers to only learn the residual modification F\mathcal{F}. If the identity mapping is optimal, the weights can simply be driven to zero, which is easier than learning an identity mapping from scratch using stacked non-linear layers.

Sources

S3.E1

𝐲=ℱ​(𝐱,{Wi})+𝐱.𝐲ℱ𝐱subscript𝑊𝑖𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}. (1)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}
Implementation detail

Illustrative Numerical Example

Let us compute the output y\mathbf{y} for a single-channel 2D vector input x\mathbf{x} and a simple linear residual function F(x,W)=Wx\mathcal{F}(\mathbf{x}, W) = W \mathbf{x}.

Given:

  • Input vector: x=[1.02.0]\mathbf{x} = \begin{bmatrix} 1.0 \\ 2.0 \end{bmatrix}
  • Weight matrix: W=[0.10.20.30.4]W = \begin{bmatrix} 0.1 & 0.2 \\ 0.3 & 0.4 \end{bmatrix}

Step-by-step calculation:

  1. Compute the residual mapping F(x,W)=Wx\mathcal{F}(\mathbf{x}, W) = W \mathbf{x}: F(x,W)=[0.10.20.30.4][1.02.0]=[(0.1×1.0)+(0.2×2.0)(0.3×1.0)+(0.4×2.0)]=[0.51.1]\mathcal{F}(\mathbf{x}, W) = \begin{bmatrix} 0.1 & 0.2 \\ 0.3 & 0.4 \end{bmatrix} \begin{bmatrix} 1.0 \\ 2.0 \end{bmatrix} = \begin{bmatrix} (0.1 \times 1.0) + (0.2 \times 2.0) \\ (0.3 \times 1.0) + (0.4 \times 2.0) \end{bmatrix} = \begin{bmatrix} 0.5 \\ 1.1 \end{bmatrix}
  2. Add the identity shortcut connection x\mathbf{x}: y=F(x,W)+x=[0.51.1]+[1.02.0]=[1.53.1]\mathbf{y} = \mathcal{F}(\mathbf{x}, W) + \mathbf{x} = \begin{bmatrix} 0.5 \\ 1.1 \end{bmatrix} + \begin{bmatrix} 1.0 \\ 2.0 \end{bmatrix} = \begin{bmatrix} 1.5 \\ 3.1 \end{bmatrix}
Sources

S3.E1

𝐲=ℱ​(𝐱,{Wi})+𝐱.𝐲ℱ𝐱subscript𝑊𝑖𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}. (1)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}
y\mathbf{y}
Output vector of the residual block · vector
x\mathbf{x}
Input vector to the residual block · vector
F\mathcal{F}
Residual mapping function to be learned by the stacked layers · function
{Wi}\{W_{i}\}
Set of weights associated with the stacked layers in the block · set of matrices