Study the paper
Deep Residual Learning for Image Recognition
Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.
The Degradation Problem and Optimization Difficulties
The Degradation Problem and Residual Learning Context
The Degradation Problem and Residual Learning Context
The Degradation Problem in Deep Networks
Sources
S1.F1
Figure 1: Training error (left) and test error (right) on CIFAR-10 with 20-layer and 56-layer “plain” networks. The deeper network has higher training error, and thus test error. Similar phenomena on ImageNet is presented in Fig. 4.
S1.p3.1
When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error, as reported in [11, 42] and thoroughly verified by our experiments. Fig. 1 shows a typical example.
As network depth increases, a degradation problem is exposed: accuracy saturates and then degrades rapidly. Crucially, this degradation is not caused by overfitting, because the training error itself increases when more layers are added to a sufficiently deep model. If overfitting were the cause, we would observe low training error but high test error; instead, both training and test errors degrade.
Sources
S1.p3.1
When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error, as reported in [11, 42] and thoroughly verified by our experiments. Fig. 1 shows a typical example.
Deep dive
To understand why this is an optimization failure, consider a shallower architecture and its deeper counterpart. Theoretically, a deeper model has a solution by construction: copy the learned layers from the shallower model and set the additional layers to perform identity mappings. This constructed solution guarantees that the deeper model should produce no higher training error than its shallower counterpart. The fact that solvers fail to find such a solution indicates that optimization becomes significantly harder with increased depth.
Sources
S1.p4.1
The degradation (of training accuracy) indicates that not all systems are similarly easy to optimize. Let us consider a shallower architecture and its deeper counterpart that adds more layers onto it. There exists a solution by construction to the deeper model: the added layers are identity mapping, and the other layers are copied from the learned shallower model. The existence of this constructed solution indicates that a deeper model should produce no higher training error than its shallower counterpart. But experiments show that our current solvers on hand are unable to find solutions that are comparably good or better than the constructed solution (or unable to do so in feasible time).
Formulating Residual Learning
Sources
S1.p5.3
In this paper, we address the degradation problem by introducing a deep residual learning framework. Instead of hoping each few stacked layers directly fit a desired underlying mapping, we explicitly let these layers fit a residual mapping. Formally, denoting the desired underlying mapping as ℋ(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}), we let the stacked nonlinear layers fit another mapping of ℱ(𝐱):=ℋ(𝐱)−𝐱assignℱ𝐱ℋ𝐱𝐱\mathcal{F}(\mathbf{x}):=\mathcal{H}(\mathbf{x})-\mathbf{x}. The original mapping is recast into ℱ(𝐱)+𝐱ℱ𝐱𝐱\mathcal{F}(\mathbf{x})+\mathbf{x}. We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping. To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers.
S3.SS1.p1.6
Let us consider ℋ(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}) as an underlying mapping to be fit by a few stacked layers (not necessarily the entire net), with 𝐱𝐱\mathbf{x} denoting the inputs to the first of these layers. If one hypothesizes that multiple nonlinear layers can asymptotically approximate complicated functions222This hypothesis, however, is still an open question. See [28]., then it is equivalent to hypothesize that they can asymptotically approximate the residual functions, i.e., ℋ(𝐱)−𝐱ℋ𝐱𝐱\mathcal{H}(\mathbf{x})-\mathbf{x} (assuming that the input and output are of the same dimensions). So rather than expect stacked layers to approximate ℋ(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}), we explicitly let these layers approximate a residual function ℱ(𝐱):=ℋ(𝐱)−𝐱assignℱ𝐱ℋ𝐱𝐱\mathcal{F}(\mathbf{x}):=\mathcal{H}(\mathbf{x})-\mathbf{x}. The original function thus becomes ℱ(𝐱)+𝐱ℱ𝐱𝐱\mathcal{F}(\mathbf{x})+\mathbf{x}. Although both forms should be able to asymptotically approximate the desired functions (as hypothesized), the ease of learning might be different.
To address this optimization difficulty, we reformulate the learning objective. Instead of expecting a stack of nonlinear layers to directly fit the desired underlying mapping , we let them approximate a residual mapping . The original mapping is thus recast as:
Sources
S1.p5.3
In this paper, we address the degradation problem by introducing a deep residual learning framework. Instead of hoping each few stacked layers directly fit a desired underlying mapping, we explicitly let these layers fit a residual mapping. Formally, denoting the desired underlying mapping as ℋ(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}), we let the stacked nonlinear layers fit another mapping of ℱ(𝐱):=ℋ(𝐱)−𝐱assignℱ𝐱ℋ𝐱𝐱\mathcal{F}(\mathbf{x}):=\mathcal{H}(\mathbf{x})-\mathbf{x}. The original mapping is recast into ℱ(𝐱)+𝐱ℱ𝐱𝐱\mathcal{F}(\mathbf{x})+\mathbf{x}. We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping. To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers.
S3.SS1.p1.6
Let us consider ℋ(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}) as an underlying mapping to be fit by a few stacked layers (not necessarily the entire net), with 𝐱𝐱\mathbf{x} denoting the inputs to the first of these layers. If one hypothesizes that multiple nonlinear layers can asymptotically approximate complicated functions222This hypothesis, however, is still an open question. See [28]., then it is equivalent to hypothesize that they can asymptotically approximate the residual functions, i.e., ℋ(𝐱)−𝐱ℋ𝐱𝐱\mathcal{H}(\mathbf{x})-\mathbf{x} (assuming that the input and output are of the same dimensions). So rather than expect stacked layers to approximate ℋ(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}), we explicitly let these layers approximate a residual function ℱ(𝐱):=ℋ(𝐱)−𝐱assignℱ𝐱ℋ𝐱𝐱\mathcal{F}(\mathbf{x}):=\mathcal{H}(\mathbf{x})-\mathbf{x}. The original function thus becomes ℱ(𝐱)+𝐱ℱ𝐱𝐱\mathcal{F}(\mathbf{x})+\mathbf{x}. Although both forms should be able to asymptotically approximate the desired functions (as hypothesized), the ease of learning might be different.
This formulation is hypothesized to be easier to optimize. For instance, if the optimal mapping is an identity mapping, it is much easier for optimizer solvers to push the weights of the residual mapping to zero than to learn an identity mapping from scratch using multiple stacked non-linear layers.
Sources
S1.p5.3
In this paper, we address the degradation problem by introducing a deep residual learning framework. Instead of hoping each few stacked layers directly fit a desired underlying mapping, we explicitly let these layers fit a residual mapping. Formally, denoting the desired underlying mapping as ℋ(𝐱)ℋ𝐱\mathcal{H}(\mathbf{x}), we let the stacked nonlinear layers fit another mapping of ℱ(𝐱):=ℋ(𝐱)−𝐱assignℱ𝐱ℋ𝐱𝐱\mathcal{F}(\mathbf{x}):=\mathcal{H}(\mathbf{x})-\mathbf{x}. The original mapping is recast into ℱ(𝐱)+𝐱ℱ𝐱𝐱\mathcal{F}(\mathbf{x})+\mathbf{x}. We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping. To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers.