Study the paper

Deep Residual Learning for Image Recognition

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Mathematics of Identity Shortcuts and Dimension Matching

Residual Block with Projection Shortcut

Source equation

$\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}$

This equation defines a residual block with a projection shortcut, which is used to match dimensions when the input and output dimensions of a residual block differ.

Sources

S3.E2

𝐲=ℱ(𝐱,{Wi})+Ws𝐱.𝐲ℱ𝐱subscript𝑊𝑖subscript𝑊𝑠𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)

\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}

$\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}$

Sources

S3.E2

𝐲=ℱ(𝐱,{Wi})+Ws𝐱.𝐲ℱ𝐱subscript𝑊𝑖subscript𝑊𝑠𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)

\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}

In this formulation, the residual mapping $\mathcal{F}(\mathbf{x},\{W_{i}\})$ is added to a linear projection of the input $W_{s}\mathbf{x}$ . The projection matrix $W_s$ is used to project the channel dimension of the input $\mathbf{x}$ to match the channel dimension of the output $\mathbf{y}$ .

Sources

S3.E2

𝐲=ℱ(𝐱,{Wi})+Ws𝐱.𝐲ℱ𝐱subscript𝑊𝑖subscript𝑊𝑠𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)

\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}

Implementation detail

Illustrative Calculation

Let us consider a simple deterministic example where:

Input vector $\mathbf{x} = \begin{bmatrix} 1 \\ 2 \end{bmatrix}$ (dimension $D_{in} = 2$ )
Residual mapping output $\mathcal{F}(\mathbf{x}, \{W_i\}) = \begin{bmatrix} 0.5 \\ -1.5 \\ 2.0 \end{bmatrix}$ (dimension $D_{out} = 3$ )
Projection matrix $W_s = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \end{bmatrix}$ (dimension $3 \times 2$ )

We calculate the projected shortcut: $W_s \mathbf{x} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \end{bmatrix} \begin{bmatrix} 1 \\ 2 \end{bmatrix} = \begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix}$

Then, we add the residual mapping output to obtain the final output $\mathbf{y}$ : $\mathbf{y} = \begin{bmatrix} 0.5 \\ -1.5 \\ 2.0 \end{bmatrix} + \begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix} = \begin{bmatrix} 1.5 \\ 0.5 \\ 5.0 \end{bmatrix}$

Sources

S3.E2

𝐲=ℱ(𝐱,{Wi})+Ws𝐱.𝐲ℱ𝐱subscript𝑊𝑖subscript𝑊𝑠𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)

\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}

$\mathbf{y}$: Output vector of the residual block · [D_{out}]
$\mathbf{x}$: Input vector to the residual block · [D_{in}]
$\mathcal{F}$: Residual mapping function to be learned · Function mapping [D_{in}] to [D_{out}]
$\{W_{i}\}$: Set of weights associated with the residual layers · Set of weight matrices
$W_{s}$: Linear projection matrix used to match dimensions · [D_{out}, D_{in}]