Study the paper
Deep Residual Learning for Image Recognition
Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.
Empirical Analysis and Layer Responses
Empirical Analysis and Extreme Depth Behavior
Empirical Analysis and Extreme Depth Behavior
Empirical Analysis of Layer Responses
Sources
S4.SS2.p7.1
Analysis of Layer Responses. Fig. 7 shows the standard deviations (std) of the layer responses. The responses are the outputs of each 3×\times3 layer, after BN and before other nonlinearity (ReLU/addition). For ResNets, this analysis reveals the response strength of the residual functions. Fig. 7 shows that ResNets have generally smaller responses than their plain counterparts. These results support our basic motivation (Sec.3.1) that the residual functions might be generally closer to zero than the non-residual functions. We also notice that the deeper ResNet has smaller magnitudes of responses, as evidenced by the comparisons among ResNet-20, 56, and 110 in Fig. 7. When there are more layers, an individual layer of ResNets tends to modify the signal less.
An analysis of the standard deviations (std) of layer responses on CIFAR-10 reveals key insights into how ResNets function. The layer responses are measured at the outputs of each layer, after Batch Normalization (BN) and before other non-linearities (ReLU or addition). The empirical results show that:
- Smaller Response Magnitudes: ResNets generally exhibit smaller response standard deviations compared to their plain counterparts. This supports the core hypothesis that residual functions are generally closer to zero than non-residual functions.
- Depth and Response Strength: Deeper ResNets (e.g., ResNet-56 and ResNet-110) have even smaller response magnitudes than shallower ones (e.g., ResNet-20). As the network depth increases, individual layers tend to modify the signal less, making incremental adjustments rather than drastic transformations.
Sources
S4.SS2.p7.1
Analysis of Layer Responses. Fig. 7 shows the standard deviations (std) of the layer responses. The responses are the outputs of each 3×\times3 layer, after BN and before other nonlinearity (ReLU/addition). For ResNets, this analysis reveals the response strength of the residual functions. Fig. 7 shows that ResNets have generally smaller responses than their plain counterparts. These results support our basic motivation (Sec.3.1) that the residual functions might be generally closer to zero than the non-residual functions. We also notice that the deeper ResNet has smaller magnitudes of responses, as evidenced by the comparisons among ResNet-20, 56, and 110 in Fig. 7. When there are more layers, an individual layer of ResNets tends to modify the signal less.
S4.F7
Figure 7: Standard deviations (std) of layer responses on CIFAR-10. The responses are the outputs of each 3×\times3 layer, after BN and before nonlinearity. Top: the layers are shown in their original order. Bottom: the responses are ranked in descending order.
Behavior of Aggressively Deep Networks (1202-Layer ResNet)
Sources
S4.SS2.p8.3
Exploring Over 1000 layers. We explore an aggressively deep model of over 1000 layers. We set n=200𝑛200n=200 that leads to a 1202-layer network, which is trained as described above. Our method shows no optimization difficulty, and this 103superscript10310^{3}-layer network is able to achieve training error <<0.1% (Fig. 6, right). Its test error is still fairly good (7.93%, Table 6).
S4.SS2.p9.1
But there are still open problems on such aggressively deep models. The testing result of this 1202-layer network is worse than that of our 110-layer network, although both have similar training error. We argue that this is because of overfitting. The 1202-layer network may be unnecessarily large (19.4M) for this small dataset. Strong regularization such as maxout [10] or dropout [14] is applied to obtain the best results ([10, 25, 24, 35]) on this dataset. In this paper, we use no maxout/dropout and just simply impose regularization via deep and thin architectures by design, without distracting from the focus on the difficulties of optimization. But combining with stronger regularization may improve results, which we will study in the future.
To push the limits of depth, the authors trained an aggressively deep 1202-layer ResNet on CIFAR-10. While this model encountered no optimization difficulties and achieved a training error of , its test error () was worse than that of the 110-layer ResNet ().
This performance gap is attributed to overfitting rather than optimization failure. The 1202-layer network is unnecessarily large (19.4M parameters) for a small dataset like CIFAR-10. Because the authors deliberately avoided strong regularization techniques (such as dropout or maxout) to isolate and study optimization behavior, the model overfit the training data. Combining such deep architectures with stronger regularization remains an open avenue for improvement.
Sources
S4.SS2.p8.3
Exploring Over 1000 layers. We explore an aggressively deep model of over 1000 layers. We set n=200𝑛200n=200 that leads to a 1202-layer network, which is trained as described above. Our method shows no optimization difficulty, and this 103superscript10310^{3}-layer network is able to achieve training error <<0.1% (Fig. 6, right). Its test error is still fairly good (7.93%, Table 6).
S4.SS2.p9.1
But there are still open problems on such aggressively deep models. The testing result of this 1202-layer network is worse than that of our 110-layer network, although both have similar training error. We argue that this is because of overfitting. The 1202-layer network may be unnecessarily large (19.4M) for this small dataset. Strong regularization such as maxout [10] or dropout [14] is applied to obtain the best results ([10, 25, 24, 35]) on this dataset. In this paper, we use no maxout/dropout and just simply impose regularization via deep and thin architectures by design, without distracting from the focus on the difficulties of optimization. But combining with stronger regularization may improve results, which we will study in the future.