Deep Residual Learning for Image Recognition

S4.SS2.p8.3

Exploring Over 1000 layers. We explore an aggressively deep model of over 1000 layers. We set n=200𝑛200n=200 that leads to a 1202-layer network, which is trained as described above. Our method shows no optimization difficulty, and this 103superscript10310^{3}-layer network is able to achieve training error <<0.1% (Fig. 6, right). Its test error is still fairly good (7.93%, Table 6).

S4.SS2.p8.3

Exploring Over 1000 layers. We explore an aggressively deep model of over 1000 layers. We set n=200𝑛200n=200 that leads to a 1202-layer network, which is trained as described above. Our method shows no optimization difficulty, and this 103superscript10310^{3}-layer network is able to achieve training error <<0.1% (Fig. 6, right). Its test error is still fairly good (7.93%, Table 6).

S4.SS2.p9.1

But there are still open problems on such aggressively deep models. The testing result of this 1202-layer network is worse than that of our 110-layer network, although both have similar training error. We argue that this is because of overfitting. The 1202-layer network may be unnecessarily large (19.4M) for this small dataset. Strong regularization such as maxout [10] or dropout [14] is applied to obtain the best results ([10, 25, 24, 35]) on this dataset. In this paper, we use no maxout/dropout and just simply impose regularization via deep and thin architectures by design, without distracting from the focus on the difficulties of optimization. But combining with stronger regularization may improve results, which we will study in the future.