Study the paper

Deep Residual Learning for Image Recognition

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Bottleneck Architectures and Layer Responses

The Bottleneck Building Block Design

Sources

S4.F5

Figure 5: A deeper residual function ℱℱ\mathcal{F} for ImageNet. Left: a building block (on 56×\times56 feature maps) as in Fig. 3 for ResNet-34. Right: a “bottleneck” building block for ResNet-50/101/152.

For deeper networks such as ResNet-50, ResNet-101, and ResNet-152, a three-layer bottleneck block is introduced to control computational complexity. This block consists of a stack of $1\times1$ , $3\times3$ , and $1\times1$ convolutions. The $1\times1$ layers are responsible for reducing and then restoring (boosting) dimensions, leaving the $3\times3$ layer as a bottleneck with smaller input/output dimensions.

Sources

S4.F5

Figure 5: A deeper residual function ℱℱ\mathcal{F} for ImageNet. Left: a building block (on 56×\times56 feature maps) as in Fig. 3 for ResNet-34. Right: a “bottleneck” building block for ResNet-50/101/152.

Empirical Analysis of Layer Responses

Sources

S4.F7

Figure 7: Standard deviations (std) of layer responses on CIFAR-10. The responses are the outputs of each 3×\times3 layer, after BN and before nonlinearity. Top: the layers are shown in their original order. Bottom: the responses are ranked in descending order.

Empirical results on CIFAR-10 show that residual networks generally have smaller standard deviations of layer responses compared to their plain counterparts. This supports the hypothesis that residual functions are generally closer to zero than non-residual functions, indicating that the identity mapping provides a strong and stable baseline.

Sources

S4.F7

Figure 7: Standard deviations (std) of layer responses on CIFAR-10. The responses are the outputs of each 3×\times3 layer, after BN and before nonlinearity. Top: the layers are shown in their original order. Bottom: the responses are ranked in descending order.