You are reading immutable version 4. The current guide may be newer.

Study the paper

Deep Residual Learning for Image Recognition

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Dimension Matching and Projection Shortcuts

Residual Building Block with Projection Shortcut

Residual Building Block with Projection Shortcut

Source equation

y=F(x,{Wi})+Wsx\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}

This equation defines a residual building block with a projection shortcut. When the input dimension of x\mathbf{x} differs from the output dimension of the residual function F\mathcal{F}, a linear projection matrix WsW_s is applied to the shortcut connection to match the dimensions.

Sources

S3.E2

𝐲=ℱ​(𝐱,{Wi})+Ws​𝐱.𝐲ℱ𝐱subscript𝑊𝑖subscript𝑊𝑠𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}

y=F(x,{Wi})+Wsx\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}

Sources

S3.E2

𝐲=ℱ​(𝐱,{Wi})+Ws​𝐱.𝐲ℱ𝐱subscript𝑊𝑖subscript𝑊𝑠𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}

Illustrative Calculation

Let us compute a deterministic example where the input vector x\mathbf{x} is 2-dimensional and the output y\mathbf{y} is 3-dimensional.

  • Input vector: x=[12]\mathbf{x} = \begin{bmatrix} 1 \\ 2 \end{bmatrix}
  • Residual function output: F(x,{Wi})=[0.5−1.01.5]\mathcal{F}(\mathbf{x}, \{W_i\}) = \begin{bmatrix} 0.5 \\ -1.0 \\ 1.5 \end{bmatrix}
  • Projection matrix: Ws=[100111]W_s = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \end{bmatrix}

First, we compute the projection shortcut WsxW_s \mathbf{x}: Wsx=[100111][12]=[123]W_s \mathbf{x} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \end{bmatrix} \begin{bmatrix} 1 \\ 2 \end{bmatrix} = \begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix}

Next, we add the residual function output to the projected shortcut: y=[0.5−1.01.5]+[123]=[1.51.04.5]\mathbf{y} = \begin{bmatrix} 0.5 \\ -1.0 \\ 1.5 \end{bmatrix} + \begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix} = \begin{bmatrix} 1.5 \\ 1.0 \\ 4.5 \end{bmatrix}

Sources

S3.E2

𝐲=ℱ​(𝐱,{Wi})+Ws​𝐱.𝐲ℱ𝐱subscript𝑊𝑖subscript𝑊𝑠𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}
y\mathbf{y}
Output vector of the residual block · [D_{out}]
x\mathbf{x}
Input vector to the residual block · [D_{in}]
F\mathcal{F}
Residual mapping function to be learned · [D_{out}]
{Wi}\{W_{i}\}
Set of weight matrices associated with the residual layers · Set of matrices
WsW_{s}
Linear projection matrix used to match dimensions of the shortcut connection to the output · [D_{out}, D_{in}]