You are reading immutable version 1. The current guide may be newer.

Study the paper

Deep Residual Learning for Image Recognition

Lessons, visuals, quizzes, flashcards, and resourcesβ€”organized in teaching order.

All activities

Mathematical Formulation of Residual Learning

Residual Block with Identity Shortcut

Residual Block with Identity Shortcut

Source equation

y=F(x,{Wi})+x.\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}.

This equation defines the fundamental residual block with an identity shortcut connection. The input vector is directly added to the output of the residual function, allowing gradients to flow unimpeded during backpropagation.

Sources

S3.E1

𝐲=ℱ​(𝐱,{Wi})+𝐱.𝐲ℱ𝐱subscriptπ‘Šπ‘–π±\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}. (1)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}.

y=F(x,{Wi})+x.\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}.

Sources

S3.E1

𝐲=ℱ​(𝐱,{Wi})+𝐱.𝐲ℱ𝐱subscriptπ‘Šπ‘–π±\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}. (1)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}.
Deep dive

Let us perform a deterministic calculation to illustrate this identity mapping.

Assume a 2-dimensional input vector: x=[1.0βˆ’2.0]\mathbf{x} = \begin{bmatrix} 1.0 \\ -2.0 \end{bmatrix}

Suppose the residual mapping F(x,{Wi})\mathcal{F}(\mathbf{x}, \{W_i\}) computes a simple linear transformation followed by a ReLU activation: F(x,{Wi})=ReLU(W1x)\mathcal{F}(\mathbf{x}, \{W_i\}) = \text{ReLU}(W_1 \mathbf{x}) where W1=[2.01.00.5βˆ’1.0]W_1 = \begin{bmatrix} 2.0 & 1.0 \\ 0.5 & -1.0 \end{bmatrix}

First, compute the matrix-vector product: W1x=[2.0(1.0)+1.0(βˆ’2.0)0.5(1.0)+(βˆ’1.0)(βˆ’2.0)]=[0.02.5]W_1 \mathbf{x} = \begin{bmatrix} 2.0(1.0) + 1.0(-2.0) \\ 0.5(1.0) + (-1.0)(-2.0) \end{bmatrix} = \begin{bmatrix} 0.0 \\ 2.5 \end{bmatrix}

Applying the element-wise ReLU activation: F(x,{Wi})=ReLU([0.02.5])=[0.02.5]\mathcal{F}(\mathbf{x}, \{W_i\}) = \text{ReLU}\left(\begin{bmatrix} 0.0 \\ 2.5 \end{bmatrix}\right) = \begin{bmatrix} 0.0 \\ 2.5 \end{bmatrix}

Finally, add the identity shortcut connection x\mathbf{x}: y=F(x,{Wi})+x=[0.02.5]+[1.0βˆ’2.0]=[1.00.5]\mathbf{y} = \mathcal{F}(\mathbf{x}, \{W_i\}) + \mathbf{x} = \begin{bmatrix} 0.0 \\ 2.5 \end{bmatrix} + \begin{bmatrix} 1.0 \\ -2.0 \end{bmatrix} = \begin{bmatrix} 1.0 \\ 0.5 \end{bmatrix}

Sources

S3.E1

𝐲=ℱ​(𝐱,{Wi})+𝐱.𝐲ℱ𝐱subscriptπ‘Šπ‘–π±\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}. (1)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}.
y\mathbf{y}
Output vector of the residual block Β· [D]
x\mathbf{x}
Input vector to the residual block Β· [D]
F\mathcal{F}
Residual mapping function to be learned Β· Function
{Wi}\{W_{i}\}
Set of weight matrices associated with the layers in the block Β· Set of Matrices