Study the paper
Deep Residual Learning for Image Recognition
Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.
8 activities
At a glance
Overview of Deep Residual LearningDeep Residual Learning Overview
LessonMathematical Formulation and Shortcut Connections
Mathematical Formulation and Shortcut ConnectionsMathematical Formulation
LessonIdentity Shortcut Connection
Mathematical Formulation and Shortcut ConnectionsThis equation defines the fundamental identity shortcut connection in Deep Residual Learning. Instead of forcing stacked layers to directly fit a desired underlying mapping, the network is reformulated to let these layers fit a residual mapping mathcal F ( mathbf x , W i ) . The original input mathbf x is added directly to the output of the residual function via a shortcut connection.
LessonProjection Shortcut Connection
Mathematical Formulation and Shortcut ConnectionsThis equation defines a residual block with a projection shortcut connection. When the input dimension of mathbf x differs from the output dimension of the residual function mathcal F , a linear projection matrix W s is applied to the input mathbf x to match the dimensions before addition.
LessonThe Bottleneck Building Block Design
ResNet Architectures and Bottleneck DesignsThe Bottleneck Design for Deeper ResNets
LessonEmpirical Analysis of Layer Responses and Extreme Depth
Empirical Analysis and Layer ResponsesEmpirical Analysis of Layer Responses
LessonStandard deviations of layer responses on CIFAR-10
Empirical Analysis and Layer ResponsesAn analysis of the standard deviations (std) of layer responses on CIFAR-10 is presented in Figure 7. The responses are measured at the outputs of each 3 times3 layer, after Batch Normalization (BN) and before the nonlinearity. The results show that residual networks (ResNets) generally have smaller response standard deviations compared to their plain counterparts, supporting the hypothesis that residual functions are closer to zero than non-residual functions.
QuizTest your understanding
Comprehensive Assessment14 questions grounded in this paper section.