Study the paper
Deep Residual Learning for Image Recognition
Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.
Introduction to Residual Learning
At a glance
At a glance
Introduction to Residual Learning
Sources
S1.p3.1
When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error, as reported in [11, 42] and thoroughly verified by our experiments. Fig. 1 shows a typical example.
As neural networks become deeper, they expose a degradation problem: beyond a certain depth, accuracy saturates and then degrades rapidly. This degradation is not caused by overfitting, as adding more layers to a sufficiently deep model actually leads to higher training error.
Sources
S1.p3.1
When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error, as reported in [11, 42] and thoroughly verified by our experiments. Fig. 1 shows a typical example.
To address this, the residual learning framework reformulates the objective of stacked layers. Instead of expecting the layers to directly fit a desired underlying mapping , we let them fit a residual mapping . The original mapping is then reconstructed as . This formulation makes optimization significantly easier, especially when identity mappings are close to optimal.
Sources
S1.p5.3
In this paper, we address the degradation problem by introducing a deep residual learning framework. Instead of hoping each few stacked layers directly fit a desired underlying mapping, we explicitly let these layers fit a residual mapping. Formally, denoting the desired underlying mapping as ℋ(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}), we let the stacked nonlinear layers fit another mapping of ℱ(𝐱):=ℋ(𝐱)−𝐱assignℱ𝐱ℋ𝐱𝐱\mathcal{F}(\mathbf{x}):=\mathcal{H}(\mathbf{x})-\mathbf{x}. The original mapping is recast into ℱ(𝐱)+𝐱ℱ𝐱𝐱\mathcal{F}(\mathbf{x})+\mathbf{x}. We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping. To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers.