You are reading immutable version 1. The current guide may be newer.

Study the paper

Deep Residual Learning for Image Recognition

Lessons, visuals, quizzes, flashcards, and resourcesβ€”organized in teaching order.

All activities

Identity vs. Projection Shortcuts

Residual Block with Projection Shortcut

Residual Block with Projection Shortcut

Source equation

y=F(x,{Wi})+Wsx.\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}.

When the dimensions of the input x\mathbf{x} and the residual output F\mathcal{F} differ (for example, when changing channel depths), a projection shortcut WsW_s is applied to the input to match the dimensions.

Sources

S3.E2

𝐲=ℱ​(𝐱,{Wi})+Ws​𝐱.𝐲ℱ𝐱subscriptπ‘Šπ‘–subscriptπ‘Šπ‘ π±\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}.

y=F(x,{Wi})+Wsx.\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}.

Sources

S3.E2

𝐲=ℱ​(𝐱,{Wi})+Ws​𝐱.𝐲ℱ𝐱subscriptπ‘Šπ‘–subscriptπ‘Šπ‘ π±\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}.
Deep dive

Let us perform a deterministic calculation to illustrate the projection shortcut.

Assume a 2-dimensional input vector: x=[2.04.0]\mathbf{x} = \begin{bmatrix} 2.0 \\ 4.0 \end{bmatrix}

Suppose the residual mapping F(x,{Wi})\mathcal{F}(\mathbf{x}, \{W_i\}) projects the input to a 3-dimensional space: F(x,{Wi})=[1.00.0βˆ’1.0]\mathcal{F}(\mathbf{x}, \{W_i\}) = \begin{bmatrix} 1.0 \\ 0.0 \\ -1.0 \end{bmatrix}

To perform the addition, we must project the 2-dimensional input x\mathbf{x} to 3 dimensions using the projection matrix WsW_s: Ws=[1.50.00.00.51.0βˆ’1.0]W_s = \begin{bmatrix} 1.5 & 0.0 \\ 0.0 & 0.5 \\ 1.0 & -1.0 \end{bmatrix}

Compute the projected shortcut: Wsx=[1.5(2.0)+0.0(4.0)0.0(2.0)+0.5(4.0)1.0(2.0)+(βˆ’1.0)(4.0)]=[3.02.0βˆ’2.0]W_s \mathbf{x} = \begin{bmatrix} 1.5(2.0) + 0.0(4.0) \\ 0.0(2.0) + 0.5(4.0) \\ 1.0(2.0) + (-1.0)(4.0) \end{bmatrix} = \begin{bmatrix} 3.0 \\ 2.0 \\ -2.0 \end{bmatrix}

Finally, add the residual mapping and the projected input: y=F(x,{Wi})+Wsx=[1.00.0βˆ’1.0]+[3.02.0βˆ’2.0]=[4.02.0βˆ’3.0]\mathbf{y} = \mathcal{F}(\mathbf{x}, \{W_i\}) + W_s \mathbf{x} = \begin{bmatrix} 1.0 \\ 0.0 \\ -1.0 \end{bmatrix} + \begin{bmatrix} 3.0 \\ 2.0 \\ -2.0 \end{bmatrix} = \begin{bmatrix} 4.0 \\ 2.0 \\ -3.0 \end{bmatrix}

Sources

S3.E2

𝐲=ℱ​(𝐱,{Wi})+Ws​𝐱.𝐲ℱ𝐱subscriptπ‘Šπ‘–subscriptπ‘Šπ‘ π±\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}.
y\mathbf{y}
Output vector of the residual block Β· [D_out]
x\mathbf{x}
Input vector to the residual block Β· [D_in]
F\mathcal{F}
Residual mapping function to be learned Β· Function
{Wi}\{W_{i}\}
Set of weight matrices associated with the layers in the block Β· Set of Matrices
WsW_{s}
Linear projection matrix used to match dimensions Β· [D_out, D_in]