Study the paper
Deep Residual Learning for Image Recognition
Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.
14 activities
At a glance
Course Overview: Deep Residual LearningCourse Overview: Deep Residual Learning
LessonUnderstanding the Degradation Problem
The Degradation Problem and Optimization DifficultiesThe Degradation Problem in Deep Networks
LessonOverfitting vs. Degradation
The Degradation Problem and Optimization DifficultiesIn classical machine learning, a model with more parameters typically risks overfitting, which is characterized by extremely low training error but high test error. The degradation problem is distinct because the training error itself increases, indicating that the optimization algorithm is failing to converge to a suitable parameter configuration.
LessonMathematical Formulation of Residual Learning
Mathematical Formulation of Residual LearningResidual Learning Formulation
LessonResidual Building Block with Identity Mapping
Mathematical Formulation of Residual LearningThis equation defines the fundamental building block of Deep Residual Learning (ResNet). Instead of forcing stacked layers to directly fit a desired underlying mapping mathcal H ( mathbf x ) , we let these layers approximate a residual mapping mathcal F ( mathbf x , W i ) = mathcal H ( mathbf x ) - mathbf x . The original mapping is recast into mathcal F ( mathbf x ) + mathbf x via an identity shortcut connection.
VisualResidual Building Block with Identity Mapping: step by step
Mathematical Formulation of Residual LearningFollow the existing bounded walkthrough in its intended sequence.
LessonZero-Padding vs. Projection Shortcuts
Dimension Matching and Projection ShortcutsDimension Matching in Shortcut Connections
LessonResidual Building Block with Projection Shortcut
Dimension Matching and Projection ShortcutsThis equation defines a residual building block with a projection shortcut. When the input dimension of mathbf x differs from the output dimension of the residual function mathcal F , a linear projection matrix W s is applied to the shortcut connection to match the dimensions.
VisualResidual Building Block with Projection Shortcut: step by step
Dimension Matching and Projection ShortcutsFollow the existing bounded walkthrough in its intended sequence.
LessonThe Bottleneck Building Block
Deeper Bottleneck ArchitecturesDeeper Bottleneck Architectures
LessonEmpirical Analysis of Layer Responses
Empirical Analysis of Layer ResponsesAn analysis of the standard deviations (std) of layer responses on CIFAR-10 reveals that ResNets generally have smaller responses than their plain counterparts. This supports the motivation that residual functions are closer to zero than non-residual functions. Furthermore, deeper ResNets (such as ResNet-56 and ResNet-110 compared to ResNet-20) exhibit even smaller response magnitudes, indicating that individual layers modify the signal less as depth increases.
QuizTest your understanding
Comprehensive Assessment12 questions grounded in this paper section.
ResourceFurther learning
Comprehensive Assessment3 supplementary resources for this paper section.
ResourceResearch and implementation context
Comprehensive AssessmentExplore notable related work and public implementation context.