Study the paper

Deep Residual Learning for Image Recognition

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

ResNet Architectures and Bottleneck Designs

The Bottleneck Building Block Design

The Bottleneck Design for Deeper ResNets

Sources

S4.SS1.p11.6

Deeper Bottleneck Architectures. Next we describe our deeper nets for ImageNet. Because of concerns on the training time that we can afford, we modify the building block as a bottleneck design444Deeper non-bottleneck ResNets (e.g., Fig. 5 left) also gain accuracy from increased depth (as shown on CIFAR-10), but are not as economical as the bottleneck ResNets. So the usage of bottleneck designs is mainly due to practical considerations. We further note that the degradation problem of plain nets is also witnessed for the bottleneck designs.. For each residual function ℱℱ\mathcal{F}, we use a stack of 3 layers instead of 2 (Fig. 5). The three layers are 1×\times1, 3×\times3, and 1×\times1 convolutions, where the 1×\times1 layers are responsible for reducing and then increasing (restoring) dimensions, leaving the 3×\times3 layer a bottleneck with smaller input/output dimensions. Fig. 5 shows an example, where both designs have similar time complexity.

To scale residual networks to extreme depths (such as ResNet-50, 101, and 152) while maintaining manageable training times and computational budgets, the authors introduced a modified building block known as the bottleneck design. Instead of stacking two $3\times3$ convolutional layers, this design uses a three-layer stack of $1\times1$ , $3\times3$ , and $1\times1$ convolutions.

Sources

S4.SS1.p11.6

Deeper Bottleneck Architectures. Next we describe our deeper nets for ImageNet. Because of concerns on the training time that we can afford, we modify the building block as a bottleneck design444Deeper non-bottleneck ResNets (e.g., Fig. 5 left) also gain accuracy from increased depth (as shown on CIFAR-10), but are not as economical as the bottleneck ResNets. So the usage of bottleneck designs is mainly due to practical considerations. We further note that the degradation problem of plain nets is also witnessed for the bottleneck designs.. For each residual function ℱℱ\mathcal{F}, we use a stack of 3 layers instead of 2 (Fig. 5). The three layers are 1×\times1, 3×\times3, and 1×\times1 convolutions, where the 1×\times1 layers are responsible for reducing and then increasing (restoring) dimensions, leaving the 3×\times3 layer a bottleneck with smaller input/output dimensions. Fig. 5 shows an example, where both designs have similar time complexity.

Deep dive

The roles of the three layers in the bottleneck block are as follows:

First $1\times1$ Convolution: Responsible for reducing the dimensionality (channel depth) of the input. This creates a "bottleneck" with smaller dimensions.
$3\times3$ Convolution: Performs spatial feature extraction on the reduced-dimensional representation, significantly reducing the computational cost compared to running a $3\times3$ convolution on the full input dimension.
Second $1\times1$ Convolution: Restores the dimensions back to the original high-dimensional state so that the output can be added to the identity shortcut.

Sources

S4.SS1.p11.6

Deeper Bottleneck Architectures. Next we describe our deeper nets for ImageNet. Because of concerns on the training time that we can afford, we modify the building block as a bottleneck design444Deeper non-bottleneck ResNets (e.g., Fig. 5 left) also gain accuracy from increased depth (as shown on CIFAR-10), but are not as economical as the bottleneck ResNets. So the usage of bottleneck designs is mainly due to practical considerations. We further note that the degradation problem of plain nets is also witnessed for the bottleneck designs.. For each residual function ℱℱ\mathcal{F}, we use a stack of 3 layers instead of 2 (Fig. 5). The three layers are 1×\times1, 3×\times3, and 1×\times1 convolutions, where the 1×\times1 layers are responsible for reducing and then increasing (restoring) dimensions, leaving the 3×\times3 layer a bottleneck with smaller input/output dimensions. Fig. 5 shows an example, where both designs have similar time complexity.

The parameter-free identity shortcut is particularly crucial for bottleneck architectures. If the identity shortcut were replaced with a projection shortcut (which uses a $1\times1$ convolution to match dimensions), the time complexity and model size would double because the shortcut would connect the two high-dimensional ends of the block. Using parameter-free identity shortcuts ensures that bottleneck designs remain highly efficient.

Sources

S4.SS1.p12.1

The parameter-free identity shortcuts are particularly important for the bottleneck architectures. If the identity shortcut in Fig. 5 (right) is replaced with projection, one can show that the time complexity and model size are doubled, as the shortcut is connected to the two high-dimensional ends. So identity shortcuts lead to more efficient models for the bottleneck designs.