You are reading immutable version 2. The current guide may be newer.

Study the paper

Deep Residual Learning for Image Recognition

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Mathematical Formulation and Shortcut Connections

Projection Shortcut Connection

Projection Shortcut Connection

Source equation

y=F(x,{Wi})+Wsx.\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}.

This equation defines a residual block with a projection shortcut connection. When the input dimension of x\mathbf{x} differs from the output dimension of the residual function F\mathcal{F}, a linear projection matrix WsW_s is applied to the input x\mathbf{x} to match the dimensions before addition.

Sources

S3.E2

𝐲=ℱ​(𝐱,{Wi})+Ws​𝐱.𝐲ℱ𝐱subscript𝑊𝑖subscript𝑊𝑠𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}.

y=F(x,{Wi})+Wsx.\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}.

Sources

S3.E2

𝐲=ℱ​(𝐱,{Wi})+Ws​𝐱.𝐲ℱ𝐱subscript𝑊𝑖subscript𝑊𝑠𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}.

Illustrative Toy Calculation

Let us consider a simple 1D-to-2D projection shortcut:

  • Input vector: x=[3.0]\mathbf{x} = [3.0]
  • Residual mapping output: F(x,{Wi})=[1.5,−0.5]T\mathcal{F}(\mathbf{x}, \{W_i\}) = [1.5, -0.5]^T
  • Projection matrix: Ws=[2.01.0]W_s = \begin{bmatrix} 2.0 \\ 1.0 \end{bmatrix}

Step 1: Compute the projection shortcut Wsx=[2.01.0][3.0]=[6.03.0]W_s \mathbf{x} = \begin{bmatrix} 2.0 \\ 1.0 \end{bmatrix} [3.0] = \begin{bmatrix} 6.0 \\ 3.0 \end{bmatrix}

Step 2: Add the residual mapping output y=[1.5−0.5]+[6.03.0]=[7.52.5]\mathbf{y} = \begin{bmatrix} 1.5 \\ -0.5 \end{bmatrix} + \begin{bmatrix} 6.0 \\ 3.0 \end{bmatrix} = \begin{bmatrix} 7.5 \\ 2.5 \end{bmatrix}

Sources

S3.E2

𝐲=ℱ​(𝐱,{Wi})+Ws​𝐱.𝐲ℱ𝐱subscript𝑊𝑖subscript𝑊𝑠𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}.
y\mathbf{y}
Output vector of the residual block · [D_{out}]
x\mathbf{x}
Input vector to the residual block · [D_{in}]
F\mathcal{F}
Residual mapping function to be learned · Function mapping [D_{in}] to [D_{out}]
{Wi}\{W_i\}
Set of weights associated with the residual layers · Set of matrices
WsW_s
Linear projection matrix used to match dimensions · [D_{out}, D_{in}]