Study the paper
Deep Residual Learning for Image Recognition
Lessons, visuals, quizzes, flashcards, and resourcesβorganized in teaching order.
Introduction to Residual Learning
Introduction to Residual Learning
Introduction to Residual Learning
The Degradation Problem in Deep Networks
Sources
S1.p3.1
When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error, as reported in [11, 42] and thoroughly verified by our experiments. Fig. 1 shows a typical example.
As network depth increases, accuracy saturates and then degrades rapidly. This phenomenon is known as the degradation problem. Crucially, this degradation is not caused by overfitting, because adding more layers to a sufficiently deep model leads to higher training error, not just higher test error.
Sources
S1.p3.1
When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error, as reported in [11, 42] and thoroughly verified by our experiments. Fig. 1 shows a typical example.
The Residual Learning Formulation
Sources
S1.p5.3
In this paper, we address the degradation problem by introducing a deep residual learning framework. Instead of hoping each few stacked layers directly fit a desired underlying mapping, we explicitly let these layers fit a residual mapping. Formally, denoting the desired underlying mapping as ββ(π±)βπ±\mathcal{H}(\mathbf{x}), we let the stacked nonlinear layers fit another mapping of β±β(π±):=ββ(π±)βπ±assignβ±π±βπ±π±\mathcal{F}(\mathbf{x}):=\mathcal{H}(\mathbf{x})-\mathbf{x}. The original mapping is recast into β±β(π±)+π±β±π±π±\mathcal{F}(\mathbf{x})+\mathbf{x}. We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping. To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers.
To address the degradation problem, we reformulate the mapping of stacked nonlinear layers. Instead of expecting these layers to directly fit a desired underlying mapping , we let them fit a residual mapping . The original mapping is thus recast into . It is hypothesized that optimizing this residual mapping is easier than optimizing the original, unreferenced mapping.
Sources
S1.p5.3
In this paper, we address the degradation problem by introducing a deep residual learning framework. Instead of hoping each few stacked layers directly fit a desired underlying mapping, we explicitly let these layers fit a residual mapping. Formally, denoting the desired underlying mapping as ββ(π±)βπ±\mathcal{H}(\mathbf{x}), we let the stacked nonlinear layers fit another mapping of β±β(π±):=ββ(π±)βπ±assignβ±π±βπ±π±\mathcal{F}(\mathbf{x}):=\mathcal{H}(\mathbf{x})-\mathbf{x}. The original mapping is recast into β±β(π±)+π±β±π±π±\mathcal{F}(\mathbf{x})+\mathbf{x}. We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping. To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers.