Study the paper
Deep Residual Learning for Image Recognition
Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.
Overview of Deep Residual Learning
At a glance
At a glance
Deep Residual Learning Overview
Sources
section
1 Introduction
As neural networks grow deeper, a degradation problem emerges: network accuracy saturates and then degrades rapidly. Crucially, this degradation is not caused by overfitting, as adding more layers to a sufficiently deep model leads to higher training error.
Sources
S1.p3.1
When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error, as reported in [11, 42] and thoroughly verified by our experiments. Fig. 1 shows a typical example.
To address this, the deep residual learning framework reformulates stacked layers to fit a residual mapping rather than directly fitting the desired underlying mapping. If the target mapping is denoted as , we let the stacked nonlinear layers approximate the residual mapping . The original mapping is recast as:
Sources
S1.p5.3
In this paper, we address the degradation problem by introducing a deep residual learning framework. Instead of hoping each few stacked layers directly fit a desired underlying mapping, we explicitly let these layers fit a residual mapping. Formally, denoting the desired underlying mapping as ℋ(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}), we let the stacked nonlinear layers fit another mapping of ℱ(𝐱):=ℋ(𝐱)−𝐱assignℱ𝐱ℋ𝐱𝐱\mathcal{F}(\mathbf{x}):=\mathcal{H}(\mathbf{x})-\mathbf{x}. The original mapping is recast into ℱ(𝐱)+𝐱ℱ𝐱𝐱\mathcal{F}(\mathbf{x})+\mathbf{x}. We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping. To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers.
Sources
S1.p5.3
In this paper, we address the degradation problem by introducing a deep residual learning framework. Instead of hoping each few stacked layers directly fit a desired underlying mapping, we explicitly let these layers fit a residual mapping. Formally, denoting the desired underlying mapping as ℋ(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}), we let the stacked nonlinear layers fit another mapping of ℱ(𝐱):=ℋ(𝐱)−𝐱assignℱ𝐱ℋ𝐱𝐱\mathcal{F}(\mathbf{x}):=\mathcal{H}(\mathbf{x})-\mathbf{x}. The original mapping is recast into ℱ(𝐱)+𝐱ℱ𝐱𝐱\mathcal{F}(\mathbf{x})+\mathbf{x}. We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping. To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers.
This formulation is realized in feedforward networks using shortcut connections that skip one or more layers. In the simplest case, these perform identity mapping, adding neither extra parameters nor computational complexity.
Sources
S1.p6.1
The formulation of ℱ(𝐱)+𝐱ℱ𝐱𝐱\mathcal{F}(\mathbf{x})+\mathbf{x} can be realized by feedforward neural networks with “shortcut connections” (Fig. 2). Shortcut connections [2, 34, 49] are those skipping one or more layers. In our case, the shortcut connections simply perform identity mapping, and their outputs are added to the outputs of the stacked layers (Fig. 2). Identity shortcut connections add neither extra parameter nor computational complexity. The entire network can still be trained end-to-end by SGD with backpropagation, and can be easily implemented using common libraries (e.g., Caffe [19]) without modifying the solvers.