You are reading immutable version 3. The current guide may be newer.

Study the paper

Deep Residual Learning for Image Recognition

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Mathematical Formulation of ResNets

Projection Shortcut Connection

Projection Shortcut Connection

Source equation

y=F(x,{Wi})+Wsx\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}

This equation defines the projection shortcut connection in Deep Residual Networks (ResNets). When the input dimension and output dimension of a residual block differ, a linear projection matrix WsW_s is applied to the input x\mathbf{x} to match the dimensions of the residual function F(x,{Wi})\mathcal{F}(\mathbf{x}, \{W_i\}).

Sources

S3.E2

𝐲=ℱ​(𝐱,{Wi})+Ws​𝐱.𝐲ℱ𝐱subscript𝑊𝑖subscript𝑊𝑠𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}

y=F(x,{Wi})+Wsx\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}

Sources

S3.E2

𝐲=ℱ​(𝐱,{Wi})+Ws​𝐱.𝐲ℱ𝐱subscript𝑊𝑖subscript𝑊𝑠𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}

The projection shortcut connection ensures that the dimensions of the shortcut path match the dimensions of the residual path, allowing element-wise addition.

Sources

S3.E2

𝐲=ℱ​(𝐱,{Wi})+Ws​𝐱.𝐲ℱ𝐱subscript𝑊𝑖subscript𝑊𝑠𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}
Implementation detail

Illustrative Example

Let us consider a simple deterministic example where the input vector x\mathbf{x} has dimension 2, and the output y\mathbf{y} has dimension 3.

  • Input vector: x=[1.02.0]\mathbf{x} = \begin{bmatrix} 1.0 \\ 2.0 \end{bmatrix}
  • Residual function output: F(x,{Wi})=[0.5−0.51.0]\mathcal{F}(\mathbf{x}, \{W_i\}) = \begin{bmatrix} 0.5 \\ -0.5 \\ 1.0 \end{bmatrix}
  • Projection matrix: Ws=[1.00.00.01.00.50.5]W_s = \begin{bmatrix} 1.0 & 0.0 \\ 0.0 & 1.0 \\ 0.5 & 0.5 \end{bmatrix}

Step 1: Compute the projection of the input Wsx=[1.00.00.01.00.50.5][1.02.0]=[1.02.01.5]W_s \mathbf{x} = \begin{bmatrix} 1.0 & 0.0 \\ 0.0 & 1.0 \\ 0.5 & 0.5 \end{bmatrix} \begin{bmatrix} 1.0 \\ 2.0 \end{bmatrix} = \begin{bmatrix} 1.0 \\ 2.0 \\ 1.5 \end{bmatrix}

Step 2: Add the residual function output y=F(x,{Wi})+Wsx=[0.5−0.51.0]+[1.02.01.5]=[1.51.52.5]\mathbf{y} = \mathcal{F}(\mathbf{x}, \{W_i\}) + W_s \mathbf{x} = \begin{bmatrix} 0.5 \\ -0.5 \\ 1.0 \end{bmatrix} + \begin{bmatrix} 1.0 \\ 2.0 \\ 1.5 \end{bmatrix} = \begin{bmatrix} 1.5 \\ 1.5 \\ 2.5 \end{bmatrix}

Sources

S3.E2

𝐲=ℱ​(𝐱,{Wi})+Ws​𝐱.𝐲ℱ𝐱subscript𝑊𝑖subscript𝑊𝑠𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}
y\mathbf{y}
Output vector of the residual block · [D_{out}]
x\mathbf{x}
Input vector to the residual block · [D_{in}]
F\mathcal{F}
Residual mapping function to be learned · [D_{out}]
WiW_{i}
Set of weights associated with the residual layers · variable
WsW_{s}
Linear projection matrix used to match dimensions · [D_{out}, D_{in}]