Study the paper
Deep Residual Learning for Image Recognition
Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.
12 activities
At a glance
Introduction to Residual LearningIntroduction to Residual Learning
LessonMathematical Formulation of ResNets
Mathematical Formulation of ResNetsA residual building block can be defined formally using shortcut connections. For a set of stacked layers that learn a residual mapping mathcal F ( mathbf x , W i ) , the output mathbf y is computed by adding the input mathbf x directly to the output of the stacked layers:
LessonIdentity Shortcut Connection
Mathematical Formulation of ResNetsThe fundamental formulation of deep residual learning introduces an identity shortcut connection. Instead of forcing stacked layers to directly fit a desired underlying mapping mathcal H ( mathbf x ) , we let these layers approximate a residual mapping mathcal F ( mathbf x , W i ) = mathcal H ( mathbf x ) - mathbf x . The original mapping is recast into mathcal F ( mathbf x , W i ) + mathbf x .
LessonProjection Shortcut Connection
Mathematical Formulation of ResNetsThis equation defines the projection shortcut connection in Deep Residual Networks (ResNets). When the input dimension and output dimension of a residual block differ, a linear projection matrix W s is applied to the input mathbf x to match the dimensions of the residual function mathcal F ( mathbf x , W i ) .
VisualProjection Shortcut Connection: step by step
Mathematical Formulation of ResNetsFollow the existing bounded walkthrough in its intended sequence.
VisualIdentity Shortcut Connection: step by step
Mathematical Formulation of ResNetsFollow the existing bounded walkthrough in its intended sequence.
LessonUnderstanding Bottleneck Architectures
Bottleneck Architectures for Deep ResNetsThe Bottleneck Design in Deep ResNets
LessonEmpirical Analysis and Extreme Depth Behavior
Empirical Analysis and Layer ResponsesEmpirical Analysis of Layer Responses
LessonStandard deviations of layer responses on CIFAR-10
Empirical Analysis and Layer ResponsesThe standard deviations (std) of layer responses on CIFAR-10 are analyzed to understand the behavior of residual learning. The responses are measured at the outputs of each 3 times3 layer, after Batch Normalization (BN) and before nonlinearity.
QuizTest your understanding
ResNet Knowledge Assessment12 questions grounded in this paper section.
ResourceFurther learning
ResNet Knowledge Assessment3 supplementary resources for this paper section.
ResourceResearch and implementation context
ResNet Knowledge AssessmentExplore notable related work and public implementation context.