Study the paper
Deep Residual Learning for Image Recognition
Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.
The Degradation Problem and Optimization Difficulties
Understanding the Degradation Problem
Understanding the Degradation Problem
The Degradation Problem in Deep Networks
Sources
S1.p3.1
When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error, as reported in [11, 42] and thoroughly verified by our experiments. Fig. 1 shows a typical example.
As neural network depth increases, a counterintuitive phenomenon occurs: accuracy saturates and then degrades rapidly. This is known as the degradation problem. Crucially, this degradation is not caused by overfitting, because it is characterized by higher training error alongside higher test error in deeper networks compared to shallower ones.
Sources
S1.p3.1
When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error, as reported in [11, 42] and thoroughly verified by our experiments. Fig. 1 shows a typical example.
S1.F1
Figure 1: Training error (left) and test error (right) on CIFAR-10 with 20-layer and 56-layer “plain” networks. The deeper network has higher training error, and thus test error. Similar phenomena on ImageNet is presented in Fig. 4.
Deep dive
To understand why this is an optimization failure rather than a capacity issue, consider a shallower model and a deeper counterpart constructed by adding identity mapping layers on top of it. In theory, this constructed deeper model should perform at least as well as the shallower model (producing no higher training error). However, standard optimization solvers struggle to find this or any better solution, demonstrating that deeper networks are fundamentally harder to optimize.
Sources
S1.p4.1
The degradation (of training accuracy) indicates that not all systems are similarly easy to optimize. Let us consider a shallower architecture and its deeper counterpart that adds more layers onto it. There exists a solution by construction to the deeper model: the added layers are identity mapping, and the other layers are copied from the learned shallower model. The existence of this constructed solution indicates that a deeper model should produce no higher training error than its shallower counterpart. But experiments show that our current solvers on hand are unable to find solutions that are comparably good or better than the constructed solution (or unable to do so in feasible time).