Study the paper

Deep Residual Learning for Image Recognition

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Bottleneck Architectures for Deep ResNets

Understanding Bottleneck Architectures

The Bottleneck Design in Deep ResNets

Sources

S4.SS1.p11.6

Deeper Bottleneck Architectures. Next we describe our deeper nets for ImageNet. Because of concerns on the training time that we can afford, we modify the building block as a bottleneck design444Deeper non-bottleneck ResNets (e.g., Fig. 5 left) also gain accuracy from increased depth (as shown on CIFAR-10), but are not as economical as the bottleneck ResNets. So the usage of bottleneck designs is mainly due to practical considerations. We further note that the degradation problem of plain nets is also witnessed for the bottleneck designs.. For each residual function ℱℱ\mathcal{F}, we use a stack of 3 layers instead of 2 (Fig. 5). The three layers are 1×\times1, 3×\times3, and 1×\times1 convolutions, where the 1×\times1 layers are responsible for reducing and then increasing (restoring) dimensions, leaving the 3×\times3 layer a bottleneck with smaller input/output dimensions. Fig. 5 shows an example, where both designs have similar time complexity.

To scale networks to extreme depths (such as ResNet-50, 101, and 152) while keeping training time and computational complexity manageable, the authors introduce a bottleneck building block. Instead of stacking two $3\times3$ convolutional layers as in the basic block, the bottleneck design uses a stack of three convolutional layers: a $1\times1$ convolution, a $3\times3$ convolution, and another $1\times1$ convolution.

Sources

S4.SS1.p11.6

Deeper Bottleneck Architectures. Next we describe our deeper nets for ImageNet. Because of concerns on the training time that we can afford, we modify the building block as a bottleneck design444Deeper non-bottleneck ResNets (e.g., Fig. 5 left) also gain accuracy from increased depth (as shown on CIFAR-10), but are not as economical as the bottleneck ResNets. So the usage of bottleneck designs is mainly due to practical considerations. We further note that the degradation problem of plain nets is also witnessed for the bottleneck designs.. For each residual function ℱℱ\mathcal{F}, we use a stack of 3 layers instead of 2 (Fig. 5). The three layers are 1×\times1, 3×\times3, and 1×\times1 convolutions, where the 1×\times1 layers are responsible for reducing and then increasing (restoring) dimensions, leaving the 3×\times3 layer a bottleneck with smaller input/output dimensions. Fig. 5 shows an example, where both designs have similar time complexity.

Deep dive

How the Bottleneck Reduces Complexity:

Dimension Reduction ( $1\times1$ ): The first $1\times1$ layer reduces the input feature map dimensions (channels) to a smaller bottleneck size.
Spatial Convolution ( $3\times3$ ): The $3\times3$ convolution operates on this reduced-dimensional space, significantly lowering the computational cost and parameter count compared to operating on the full input dimensions.
Dimension Restoration ( $1\times1$ ): The final $1\times1$ layer restores the dimensions back to the original high-dimensional space so that the residual mapping can be added to the identity shortcut.

Sources

S4.SS1.p11.6

Deeper Bottleneck Architectures. Next we describe our deeper nets for ImageNet. Because of concerns on the training time that we can afford, we modify the building block as a bottleneck design444Deeper non-bottleneck ResNets (e.g., Fig. 5 left) also gain accuracy from increased depth (as shown on CIFAR-10), but are not as economical as the bottleneck ResNets. So the usage of bottleneck designs is mainly due to practical considerations. We further note that the degradation problem of plain nets is also witnessed for the bottleneck designs.. For each residual function ℱℱ\mathcal{F}, we use a stack of 3 layers instead of 2 (Fig. 5). The three layers are 1×\times1, 3×\times3, and 1×\times1 convolutions, where the 1×\times1 layers are responsible for reducing and then increasing (restoring) dimensions, leaving the 3×\times3 layer a bottleneck with smaller input/output dimensions. Fig. 5 shows an example, where both designs have similar time complexity.

The parameter-free identity shortcut is critical for this architecture. If a projection shortcut (using a $1\times1$ convolution) were used instead of the identity shortcut, the shortcut would connect the two high-dimensional ends of the block. This would double both the time complexity and the model size, making the bottleneck design far less economical.

Sources

S4.SS1.p12.1

The parameter-free identity shortcuts are particularly important for the bottleneck architectures. If the identity shortcut in Fig. 5 (right) is replaced with projection, one can show that the time complexity and model size are doubled, as the shortcut is connected to the two high-dimensional ends. So identity shortcuts lead to more efficient models for the bottleneck designs.