Study the paper

Deep Residual Learning for Image Recognition

Lessons, visuals, quizzes, flashcards, and resourcesβ€”organized in teaching order.

All activities

Mathematics of Identity Shortcuts and Dimension Matching

Residual Block with Projection Shortcut

Residual Block with Projection Shortcut

Source equation

y=F(x,{Wi})+Wsx\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}

This equation defines a residual block with a projection shortcut, which is used to match dimensions when the input and output dimensions of a residual block differ.

Sources

S3.E2

𝐲=ℱ​(𝐱,{Wi})+Ws​𝐱.𝐲ℱ𝐱subscriptπ‘Šπ‘–subscriptπ‘Šπ‘ π±\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}

y=F(x,{Wi})+Wsx\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}

Sources

S3.E2

𝐲=ℱ​(𝐱,{Wi})+Ws​𝐱.𝐲ℱ𝐱subscriptπ‘Šπ‘–subscriptπ‘Šπ‘ π±\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}

In this formulation, the residual mapping F(x,{Wi})\mathcal{F}(\mathbf{x},\{W_{i}\}) is added to a linear projection of the input WsxW_{s}\mathbf{x}. The projection matrix WsW_s is used to project the channel dimension of the input x\mathbf{x} to match the channel dimension of the output y\mathbf{y}.

Sources

S3.E2

𝐲=ℱ​(𝐱,{Wi})+Ws​𝐱.𝐲ℱ𝐱subscriptπ‘Šπ‘–subscriptπ‘Šπ‘ π±\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}
Implementation detail

Illustrative Calculation

Let us consider a simple deterministic example where:

  • Input vector x=[12]\mathbf{x} = \begin{bmatrix} 1 \\ 2 \end{bmatrix} (dimension Din=2D_{in} = 2)
  • Residual mapping output F(x,{Wi})=[0.5βˆ’1.52.0]\mathcal{F}(\mathbf{x}, \{W_i\}) = \begin{bmatrix} 0.5 \\ -1.5 \\ 2.0 \end{bmatrix} (dimension Dout=3D_{out} = 3)
  • Projection matrix Ws=[100111]W_s = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \end{bmatrix} (dimension 3Γ—23 \times 2)

We calculate the projected shortcut: Wsx=[100111][12]=[123]W_s \mathbf{x} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \end{bmatrix} \begin{bmatrix} 1 \\ 2 \end{bmatrix} = \begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix}

Then, we add the residual mapping output to obtain the final output y\mathbf{y}: y=[0.5βˆ’1.52.0]+[123]=[1.50.55.0]\mathbf{y} = \begin{bmatrix} 0.5 \\ -1.5 \\ 2.0 \end{bmatrix} + \begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix} = \begin{bmatrix} 1.5 \\ 0.5 \\ 5.0 \end{bmatrix}

Sources

S3.E2

𝐲=ℱ​(𝐱,{Wi})+Ws​𝐱.𝐲ℱ𝐱subscriptπ‘Šπ‘–subscriptπ‘Šπ‘ π±\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}
y\mathbf{y}
Output vector of the residual block Β· [D_{out}]
x\mathbf{x}
Input vector to the residual block Β· [D_{in}]
F\mathcal{F}
Residual mapping function to be learned Β· Function mapping [D_{in}] to [D_{out}]
{Wi}\{W_{i}\}
Set of weights associated with the residual layers Β· Set of weight matrices
WsW_{s}
Linear projection matrix used to match dimensions Β· [D_{out}, D_{in}]