Study the paper
Deep Residual Learning for Image Recognition
Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.
The Degradation Problem and Optimization Difficulties
Overfitting vs. Degradation
Overfitting vs. Degradation
Refresher
In classical machine learning, a model with more parameters typically risks overfitting, which is characterized by extremely low training error but high test error. The degradation problem is distinct because the training error itself increases, indicating that the optimization algorithm is failing to converge to a suitable parameter configuration.
Sources
S1.p3.1
When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error, as reported in [11, 42] and thoroughly verified by our experiments. Fig. 1 shows a typical example.