Study the paper

Deep Residual Learning for Image Recognition

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Empirical Analysis and Layer Responses

Empirical Analysis of Layer Responses and Extreme Depth

Empirical Analysis of Layer Responses

Sources

S4.SS2.p7.1

Analysis of Layer Responses. Fig. 7 shows the standard deviations (std) of the layer responses. The responses are the outputs of each 3×\times3 layer, after BN and before other nonlinearity (ReLU/addition). For ResNets, this analysis reveals the response strength of the residual functions. Fig. 7 shows that ResNets have generally smaller responses than their plain counterparts. These results support our basic motivation (Sec.3.1) that the residual functions might be generally closer to zero than the non-residual functions. We also notice that the deeper ResNet has smaller magnitudes of responses, as evidenced by the comparisons among ResNet-20, 56, and 110 in Fig. 7. When there are more layers, an individual layer of ResNets tends to modify the signal less.

An analysis of the standard deviations of layer responses on CIFAR-10 provides key insights into how ResNets function. The layer responses are measured at the outputs of each $3\times3$ layer, after Batch Normalization (BN) and before other non-linearities (such as ReLU and addition). The empirical results show that ResNets have generally smaller responses than their plain counterparts. This supports the core motivation that residual functions are generally closer to zero than non-residual functions.

Sources

S4.SS2.p7.1

Analysis of Layer Responses. Fig. 7 shows the standard deviations (std) of the layer responses. The responses are the outputs of each 3×\times3 layer, after BN and before other nonlinearity (ReLU/addition). For ResNets, this analysis reveals the response strength of the residual functions. Fig. 7 shows that ResNets have generally smaller responses than their plain counterparts. These results support our basic motivation (Sec.3.1) that the residual functions might be generally closer to zero than the non-residual functions. We also notice that the deeper ResNet has smaller magnitudes of responses, as evidenced by the comparisons among ResNet-20, 56, and 110 in Fig. 7. When there are more layers, an individual layer of ResNets tends to modify the signal less.

S4.F7

Figure 7: Standard deviations (std) of layer responses on CIFAR-10. The responses are the outputs of each 3×\times3 layer, after BN and before nonlinearity. Top: the layers are shown in their original order. Bottom: the responses are ranked in descending order.

Deep dive

Furthermore, as the depth of the ResNet increases (comparing ResNet-20, 56, and 110), the magnitude of the responses becomes even smaller. This indicates that when there are more layers, an individual layer in a ResNet tends to modify the signal less, relying more on the identity shortcut to transmit information.

Sources

S4.SS2.p7.1

Analysis of Layer Responses. Fig. 7 shows the standard deviations (std) of the layer responses. The responses are the outputs of each 3×\times3 layer, after BN and before other nonlinearity (ReLU/addition). For ResNets, this analysis reveals the response strength of the residual functions. Fig. 7 shows that ResNets have generally smaller responses than their plain counterparts. These results support our basic motivation (Sec.3.1) that the residual functions might be generally closer to zero than the non-residual functions. We also notice that the deeper ResNet has smaller magnitudes of responses, as evidenced by the comparisons among ResNet-20, 56, and 110 in Fig. 7. When there are more layers, an individual layer of ResNets tends to modify the signal less.

Behavior of Aggressively Deep Models (1202-Layer ResNet)

Sources

S4.SS2.p8.3

Exploring Over 1000 layers. We explore an aggressively deep model of over 1000 layers. We set n=200𝑛200n=200 that leads to a 1202-layer network, which is trained as described above. Our method shows no optimization difficulty, and this 103superscript10310^{3}-layer network is able to achieve training error <<0.1% (Fig. 6, right). Its test error is still fairly good (7.93%, Table 6).

S4.SS2.p9.1

But there are still open problems on such aggressively deep models. The testing result of this 1202-layer network is worse than that of our 110-layer network, although both have similar training error. We argue that this is because of overfitting. The 1202-layer network may be unnecessarily large (19.4M) for this small dataset. Strong regularization such as maxout [10] or dropout [14] is applied to obtain the best results ([10, 25, 24, 35]) on this dataset. In this paper, we use no maxout/dropout and just simply impose regularization via deep and thin architectures by design, without distracting from the focus on the difficulties of optimization. But combining with stronger regularization may improve results, which we will study in the future.

To test the limits of depth, the authors trained an aggressively deep 1202-layer ResNet on CIFAR-10. While the model showed no optimization difficulties and achieved a training error of less than 0.1%, its test performance (7.93% error) was worse than that of the 110-layer ResNet. This performance drop is attributed to overfitting rather than optimization failure, as the 1202-layer model (19.4M parameters) is unnecessarily large for a small dataset like CIFAR-10 without strong regularization techniques like dropout or maxout.

Sources

S4.SS2.p8.3

Exploring Over 1000 layers. We explore an aggressively deep model of over 1000 layers. We set n=200𝑛200n=200 that leads to a 1202-layer network, which is trained as described above. Our method shows no optimization difficulty, and this 103superscript10310^{3}-layer network is able to achieve training error <<0.1% (Fig. 6, right). Its test error is still fairly good (7.93%, Table 6).

S4.SS2.p9.1

But there are still open problems on such aggressively deep models. The testing result of this 1202-layer network is worse than that of our 110-layer network, although both have similar training error. We argue that this is because of overfitting. The 1202-layer network may be unnecessarily large (19.4M) for this small dataset. Strong regularization such as maxout [10] or dropout [14] is applied to obtain the best results ([10, 25, 24, 35]) on this dataset. In this paper, we use no maxout/dropout and just simply impose regularization via deep and thin architectures by design, without distracting from the focus on the difficulties of optimization. But combining with stronger regularization may improve results, which we will study in the future.