Study the paper
Deep Residual Learning for Image Recognition
Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.
The Degradation Problem and Optimization Difficulties
The Degradation Problem and Optimization Difficulties
The Degradation Problem and Optimization Difficulties
Understanding the Degradation Problem
Sources
S1.p3.1
When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error, as reported in [11, 42] and thoroughly verified by our experiments. Fig. 1 shows a typical example.
When deeper networks begin to converge, a counterintuitive phenomenon occurs: as network depth increases, accuracy saturates and then degrades rapidly. This is referred to as the degradation problem. Crucially, this degradation is not caused by overfitting, because the deeper models exhibit higher training error than their shallower counterparts.
Sources
S1.p3.1
When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error, as reported in [11, 42] and thoroughly verified by our experiments. Fig. 1 shows a typical example.
S1.F1
Figure 1: Training error (left) and test error (right) on CIFAR-10 with 20-layer and 56-layer “plain” networks. The deeper network has higher training error, and thus test error. Similar phenomena on ImageNet is presented in Fig. 4.
To understand why this is an optimization failure, consider a shallower architecture and its deeper counterpart. In theory, there exists a solution where the added layers in the deeper model simply perform an identity mapping while copying the remaining layers from the shallower model. This constructed solution guarantees that the deeper model should have no higher training error than the shallower one. However, standard optimization solvers struggle to find this solution.
Sources
S1.p4.1
The degradation (of training accuracy) indicates that not all systems are similarly easy to optimize. Let us consider a shallower architecture and its deeper counterpart that adds more layers onto it. There exists a solution by construction to the deeper model: the added layers are identity mapping, and the other layers are copied from the learned shallower model. The existence of this constructed solution indicates that a deeper model should produce no higher training error than its shallower counterpart. But experiments show that our current solvers on hand are unable to find solutions that are comparably good or better than the constructed solution (or unable to do so in feasible time).
The Residual Learning Formulation
Sources
S1.p5.3
In this paper, we address the degradation problem by introducing a deep residual learning framework. Instead of hoping each few stacked layers directly fit a desired underlying mapping, we explicitly let these layers fit a residual mapping. Formally, denoting the desired underlying mapping as ℋ(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}), we let the stacked nonlinear layers fit another mapping of ℱ(𝐱):=ℋ(𝐱)−𝐱assignℱ𝐱ℋ𝐱𝐱\mathcal{F}(\mathbf{x}):=\mathcal{H}(\mathbf{x})-\mathbf{x}. The original mapping is recast into ℱ(𝐱)+𝐱ℱ𝐱𝐱\mathcal{F}(\mathbf{x})+\mathbf{x}. We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping. To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers.
To address this optimization difficulty, we reformulate the objective of the stacked layers. Instead of expecting the layers to directly fit a desired underlying mapping , we let them fit a residual mapping defined as:
Sources
S1.p5.3
In this paper, we address the degradation problem by introducing a deep residual learning framework. Instead of hoping each few stacked layers directly fit a desired underlying mapping, we explicitly let these layers fit a residual mapping. Formally, denoting the desired underlying mapping as ℋ(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}), we let the stacked nonlinear layers fit another mapping of ℱ(𝐱):=ℋ(𝐱)−𝐱assignℱ𝐱ℋ𝐱𝐱\mathcal{F}(\mathbf{x}):=\mathcal{H}(\mathbf{x})-\mathbf{x}. The original mapping is recast into ℱ(𝐱)+𝐱ℱ𝐱𝐱\mathcal{F}(\mathbf{x})+\mathbf{x}. We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping. To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers.
S3.SS1.p1.6
Let us consider ℋ(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}) as an underlying mapping to be fit by a few stacked layers (not necessarily the entire net), with 𝐱𝐱\mathbf{x} denoting the inputs to the first of these layers. If one hypothesizes that multiple nonlinear layers can asymptotically approximate complicated functions222This hypothesis, however, is still an open question. See [28]., then it is equivalent to hypothesize that they can asymptotically approximate the residual functions, i.e., ℋ(𝐱)−𝐱ℋ𝐱𝐱\mathcal{H}(\mathbf{x})-\mathbf{x} (assuming that the input and output are of the same dimensions). So rather than expect stacked layers to approximate ℋ(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}), we explicitly let these layers approximate a residual function ℱ(𝐱):=ℋ(𝐱)−𝐱assignℱ𝐱ℋ𝐱𝐱\mathcal{F}(\mathbf{x}):=\mathcal{H}(\mathbf{x})-\mathbf{x}. The original function thus becomes ℱ(𝐱)+𝐱ℱ𝐱𝐱\mathcal{F}(\mathbf{x})+\mathbf{x}. Although both forms should be able to asymptotically approximate the desired functions (as hypothesized), the ease of learning might be different.
Sources
S1.p5.3
In this paper, we address the degradation problem by introducing a deep residual learning framework. Instead of hoping each few stacked layers directly fit a desired underlying mapping, we explicitly let these layers fit a residual mapping. Formally, denoting the desired underlying mapping as ℋ(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}), we let the stacked nonlinear layers fit another mapping of ℱ(𝐱):=ℋ(𝐱)−𝐱assignℱ𝐱ℋ𝐱𝐱\mathcal{F}(\mathbf{x}):=\mathcal{H}(\mathbf{x})-\mathbf{x}. The original mapping is recast into ℱ(𝐱)+𝐱ℱ𝐱𝐱\mathcal{F}(\mathbf{x})+\mathbf{x}. We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping. To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers.
S3.SS1.p1.6
Let us consider ℋ(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}) as an underlying mapping to be fit by a few stacked layers (not necessarily the entire net), with 𝐱𝐱\mathbf{x} denoting the inputs to the first of these layers. If one hypothesizes that multiple nonlinear layers can asymptotically approximate complicated functions222This hypothesis, however, is still an open question. See [28]., then it is equivalent to hypothesize that they can asymptotically approximate the residual functions, i.e., ℋ(𝐱)−𝐱ℋ𝐱𝐱\mathcal{H}(\mathbf{x})-\mathbf{x} (assuming that the input and output are of the same dimensions). So rather than expect stacked layers to approximate ℋ(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}), we explicitly let these layers approximate a residual function ℱ(𝐱):=ℋ(𝐱)−𝐱assignℱ𝐱ℋ𝐱𝐱\mathcal{F}(\mathbf{x}):=\mathcal{H}(\mathbf{x})-\mathbf{x}. The original function thus becomes ℱ(𝐱)+𝐱ℱ𝐱𝐱\mathcal{F}(\mathbf{x})+\mathbf{x}. Although both forms should be able to asymptotically approximate the desired functions (as hypothesized), the ease of learning might be different.
This recasts the original mapping into:
Sources
S1.p5.3
In this paper, we address the degradation problem by introducing a deep residual learning framework. Instead of hoping each few stacked layers directly fit a desired underlying mapping, we explicitly let these layers fit a residual mapping. Formally, denoting the desired underlying mapping as ℋ(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}), we let the stacked nonlinear layers fit another mapping of ℱ(𝐱):=ℋ(𝐱)−𝐱assignℱ𝐱ℋ𝐱𝐱\mathcal{F}(\mathbf{x}):=\mathcal{H}(\mathbf{x})-\mathbf{x}. The original mapping is recast into ℱ(𝐱)+𝐱ℱ𝐱𝐱\mathcal{F}(\mathbf{x})+\mathbf{x}. We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping. To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers.
S3.SS1.p1.6
Let us consider ℋ(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}) as an underlying mapping to be fit by a few stacked layers (not necessarily the entire net), with 𝐱𝐱\mathbf{x} denoting the inputs to the first of these layers. If one hypothesizes that multiple nonlinear layers can asymptotically approximate complicated functions222This hypothesis, however, is still an open question. See [28]., then it is equivalent to hypothesize that they can asymptotically approximate the residual functions, i.e., ℋ(𝐱)−𝐱ℋ𝐱𝐱\mathcal{H}(\mathbf{x})-\mathbf{x} (assuming that the input and output are of the same dimensions). So rather than expect stacked layers to approximate ℋ(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}), we explicitly let these layers approximate a residual function ℱ(𝐱):=ℋ(𝐱)−𝐱assignℱ𝐱ℋ𝐱𝐱\mathcal{F}(\mathbf{x}):=\mathcal{H}(\mathbf{x})-\mathbf{x}. The original function thus becomes ℱ(𝐱)+𝐱ℱ𝐱𝐱\mathcal{F}(\mathbf{x})+\mathbf{x}. Although both forms should be able to asymptotically approximate the desired functions (as hypothesized), the ease of learning might be different.
Sources
S1.p5.3
In this paper, we address the degradation problem by introducing a deep residual learning framework. Instead of hoping each few stacked layers directly fit a desired underlying mapping, we explicitly let these layers fit a residual mapping. Formally, denoting the desired underlying mapping as ℋ(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}), we let the stacked nonlinear layers fit another mapping of ℱ(𝐱):=ℋ(𝐱)−𝐱assignℱ𝐱ℋ𝐱𝐱\mathcal{F}(\mathbf{x}):=\mathcal{H}(\mathbf{x})-\mathbf{x}. The original mapping is recast into ℱ(𝐱)+𝐱ℱ𝐱𝐱\mathcal{F}(\mathbf{x})+\mathbf{x}. We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping. To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers.
S3.SS1.p1.6
Let us consider ℋ(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}) as an underlying mapping to be fit by a few stacked layers (not necessarily the entire net), with 𝐱𝐱\mathbf{x} denoting the inputs to the first of these layers. If one hypothesizes that multiple nonlinear layers can asymptotically approximate complicated functions222This hypothesis, however, is still an open question. See [28]., then it is equivalent to hypothesize that they can asymptotically approximate the residual functions, i.e., ℋ(𝐱)−𝐱ℋ𝐱𝐱\mathcal{H}(\mathbf{x})-\mathbf{x} (assuming that the input and output are of the same dimensions). So rather than expect stacked layers to approximate ℋ(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}), we explicitly let these layers approximate a residual function ℱ(𝐱):=ℋ(𝐱)−𝐱assignℱ𝐱ℋ𝐱𝐱\mathcal{F}(\mathbf{x}):=\mathcal{H}(\mathbf{x})-\mathbf{x}. The original function thus becomes ℱ(𝐱)+𝐱ℱ𝐱𝐱\mathcal{F}(\mathbf{x})+\mathbf{x}. Although both forms should be able to asymptotically approximate the desired functions (as hypothesized), the ease of learning might be different.
This formulation is hypothesized to be significantly easier to optimize. For instance, if the optimal mapping is an identity mapping, it is much simpler for the solver to drive the residual weights in toward zero than to learn an identity mapping from scratch using stacked nonlinear layers.
Sources
S1.p5.3
In this paper, we address the degradation problem by introducing a deep residual learning framework. Instead of hoping each few stacked layers directly fit a desired underlying mapping, we explicitly let these layers fit a residual mapping. Formally, denoting the desired underlying mapping as ℋ(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}), we let the stacked nonlinear layers fit another mapping of ℱ(𝐱):=ℋ(𝐱)−𝐱assignℱ𝐱ℋ𝐱𝐱\mathcal{F}(\mathbf{x}):=\mathcal{H}(\mathbf{x})-\mathbf{x}. The original mapping is recast into ℱ(𝐱)+𝐱ℱ𝐱𝐱\mathcal{F}(\mathbf{x})+\mathbf{x}. We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping. To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers.
S3.SS1.p1.6
Let us consider ℋ(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}) as an underlying mapping to be fit by a few stacked layers (not necessarily the entire net), with 𝐱𝐱\mathbf{x} denoting the inputs to the first of these layers. If one hypothesizes that multiple nonlinear layers can asymptotically approximate complicated functions222This hypothesis, however, is still an open question. See [28]., then it is equivalent to hypothesize that they can asymptotically approximate the residual functions, i.e., ℋ(𝐱)−𝐱ℋ𝐱𝐱\mathcal{H}(\mathbf{x})-\mathbf{x} (assuming that the input and output are of the same dimensions). So rather than expect stacked layers to approximate ℋ(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}), we explicitly let these layers approximate a residual function ℱ(𝐱):=ℋ(𝐱)−𝐱assignℱ𝐱ℋ𝐱𝐱\mathcal{F}(\mathbf{x}):=\mathcal{H}(\mathbf{x})-\mathbf{x}. The original function thus becomes ℱ(𝐱)+𝐱ℱ𝐱𝐱\mathcal{F}(\mathbf{x})+\mathbf{x}. Although both forms should be able to asymptotically approximate the desired functions (as hypothesized), the ease of learning might be different.