You are reading immutable version 4. The current guide may be newer.

Study the paper

Deep Residual Learning for Image Recognition

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Mathematical Formulation of Residual Learning

Mathematical Formulation of Residual Learning

Mathematical Formulation of Residual Learning

Residual Learning Formulation

Sources

S1.p5.3

In this paper, we address the degradation problem by introducing a deep residual learning framework. Instead of hoping each few stacked layers directly fit a desired underlying mapping, we explicitly let these layers fit a residual mapping. Formally, denoting the desired underlying mapping as ℋ​(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}), we let the stacked nonlinear layers fit another mapping of ℱ​(𝐱):=ℋ​(𝐱)−𝐱assignℱ𝐱ℋ𝐱𝐱\mathcal{F}(\mathbf{x}):=\mathcal{H}(\mathbf{x})-\mathbf{x}. The original mapping is recast into ℱ​(𝐱)+𝐱ℱ𝐱𝐱\mathcal{F}(\mathbf{x})+\mathbf{x}. We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping. To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers.

Instead of expecting stacked layers to directly fit a desired underlying mapping H(x)\mathcal{H}(\mathbf{x}), we explicitly let these layers fit a residual mapping F(x):=H(x)−x\mathcal{F}(\mathbf{x}) := \mathcal{H}(\mathbf{x}) - \mathbf{x}. The original mapping is recast into:

Sources

S1.p5.3

In this paper, we address the degradation problem by introducing a deep residual learning framework. Instead of hoping each few stacked layers directly fit a desired underlying mapping, we explicitly let these layers fit a residual mapping. Formally, denoting the desired underlying mapping as ℋ​(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}), we let the stacked nonlinear layers fit another mapping of ℱ​(𝐱):=ℋ​(𝐱)−𝐱assignℱ𝐱ℋ𝐱𝐱\mathcal{F}(\mathbf{x}):=\mathcal{H}(\mathbf{x})-\mathbf{x}. The original mapping is recast into ℱ​(𝐱)+𝐱ℱ𝐱𝐱\mathcal{F}(\mathbf{x})+\mathbf{x}. We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping. To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers.

y=F(x,{Wi})+x\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}

Sources

S3.E1

𝐲=ℱ​(𝐱,{Wi})+𝐱.𝐲ℱ𝐱subscript𝑊𝑖𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}. (1)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}

equation

𝐲=ℱ​(𝐱,{Wi})+𝐱.𝐲ℱ𝐱subscript𝑊𝑖𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}. (1)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}

Here, x\mathbf{x} and y\mathbf{y} are the input and output vectors of the layers considered. The function F(x,{Wi})\mathcal{F}(\mathbf{x},\{W_{i}\}) represents the residual mapping to be learned. For a two-layer block, we have F=W2σ(W1x)\mathcal{F} = W_{2}\sigma(W_{1}\mathbf{x}) where σ\sigma denotes the ReLU activation function and biases are omitted for simplicity. The operation F+x\mathcal{F} + \mathbf{x} is performed by a shortcut connection and element-wise addition, followed by a second nonlinearity σ(y)\sigma(\mathbf{y}).

Sources

S3.SS2.p1.7

Here 𝐱𝐱\mathbf{x} and 𝐲𝐲\mathbf{y} are the input and output vectors of the layers considered. The function ℱ​(𝐱,{Wi})ℱ𝐱subscript𝑊𝑖\mathcal{F}(\mathbf{x},\{W_{i}\}) represents the residual mapping to be learned. For the example in Fig. 2 that has two layers, ℱ=W2​σ​(W1​𝐱)ℱsubscript𝑊2𝜎subscript𝑊1𝐱\mathcal{F}=W_{2}\sigma(W_{1}\mathbf{x}) in which σ𝜎\sigma denotes ReLU [29] and the biases are omitted for simplifying notations. The operation ℱ+𝐱ℱ𝐱\mathcal{F}+\mathbf{x} is performed by a shortcut connection and element-wise addition. We adopt the second nonlinearity after the addition (i.e., σ​(𝐲)𝜎𝐲\sigma(\mathbf{y}), see Fig. 2).