You are reading immutable version 1. The current guide may be newer.

Study the paper

Deep Residual Learning for Image Recognition

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Deeper Bottleneck Architectures

Deeper Bottleneck Architectures Walkthrough

Deeper Bottleneck Architectures Walkthrough

Deeper Bottleneck Architectures

Sources

S4.F5

Figure 5: A deeper residual function ℱℱ\mathcal{F} for ImageNet. Left: a building block (on 56×\times56 feature maps) as in Fig. 3 for ResNet-34. Right: a “bottleneck” building block for ResNet-50/101/152.

S4.SS1.p11.6

Deeper Bottleneck Architectures. Next we describe our deeper nets for ImageNet. Because of concerns on the training time that we can afford, we modify the building block as a bottleneck design444Deeper non-bottleneck ResNets (e.g., Fig. 5 left) also gain accuracy from increased depth (as shown on CIFAR-10), but are not as economical as the bottleneck ResNets. So the usage of bottleneck designs is mainly due to practical considerations. We further note that the degradation problem of plain nets is also witnessed for the bottleneck designs.. For each residual function ℱℱ\mathcal{F}, we use a stack of 3 layers instead of 2 (Fig. 5). The three layers are 1×\times1, 3×\times3, and 1×\times1 convolutions, where the 1×\times1 layers are responsible for reducing and then increasing (restoring) dimensions, leaving the 3×\times3 layer a bottleneck with smaller input/output dimensions. Fig. 5 shows an example, where both designs have similar time complexity.

For deeper networks such as ResNet-50, ResNet-101, and ResNet-152, training time and computational cost become critical constraints. To address this, the basic 2-layer residual block is modified into a 3-layer "bottleneck" design. Each residual function F\mathcal{F} uses a stack of three convolutional layers:

  1. A 1×11\times1 convolution to reduce dimensions.
  2. A 3×33\times3 convolution (the bottleneck) operating on the reduced dimensions.
  3. A 1×11\times1 convolution to restore (increase) the dimensions back to the original input size.
Sources

S4.SS1.p11.6

Deeper Bottleneck Architectures. Next we describe our deeper nets for ImageNet. Because of concerns on the training time that we can afford, we modify the building block as a bottleneck design444Deeper non-bottleneck ResNets (e.g., Fig. 5 left) also gain accuracy from increased depth (as shown on CIFAR-10), but are not as economical as the bottleneck ResNets. So the usage of bottleneck designs is mainly due to practical considerations. We further note that the degradation problem of plain nets is also witnessed for the bottleneck designs.. For each residual function ℱℱ\mathcal{F}, we use a stack of 3 layers instead of 2 (Fig. 5). The three layers are 1×\times1, 3×\times3, and 1×\times1 convolutions, where the 1×\times1 layers are responsible for reducing and then increasing (restoring) dimensions, leaving the 3×\times3 layer a bottleneck with smaller input/output dimensions. Fig. 5 shows an example, where both designs have similar time complexity.
Deep dive

The parameter-free identity shortcuts are especially critical for these bottleneck designs. If the identity shortcut were replaced with a projection shortcut, the shortcut would connect the two high-dimensional ends of the block. This would double both the time complexity and the model size. Utilizing parameter-free identity shortcuts ensures that the bottleneck design remains highly efficient and practical for extremely deep models.

Sources

S4.SS1.p12.1

The parameter-free identity shortcuts are particularly important for the bottleneck architectures. If the identity shortcut in Fig. 5 (right) is replaced with projection, one can show that the time complexity and model size are doubled, as the shortcut is connected to the two high-dimensional ends. So identity shortcuts lead to more efficient models for the bottleneck designs.