You are reading immutable version 4. The current guide may be newer.

Study the paper

Deep Residual Learning for Image Recognition

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Deeper Bottleneck Architectures

The Bottleneck Building Block

The Bottleneck Building Block

Deeper Bottleneck Architectures

Sources

S4.SS1.p11.6

Deeper Bottleneck Architectures. Next we describe our deeper nets for ImageNet. Because of concerns on the training time that we can afford, we modify the building block as a bottleneck design444Deeper non-bottleneck ResNets (e.g., Fig. 5 left) also gain accuracy from increased depth (as shown on CIFAR-10), but are not as economical as the bottleneck ResNets. So the usage of bottleneck designs is mainly due to practical considerations. We further note that the degradation problem of plain nets is also witnessed for the bottleneck designs.. For each residual function ℱℱ\mathcal{F}, we use a stack of 3 layers instead of 2 (Fig. 5). The three layers are 1×\times1, 3×\times3, and 1×\times1 convolutions, where the 1×\times1 layers are responsible for reducing and then increasing (restoring) dimensions, leaving the 3×\times3 layer a bottleneck with smaller input/output dimensions. Fig. 5 shows an example, where both designs have similar time complexity.

To make extremely deep networks (such as ResNet-50/101/152) computationally feasible, a bottleneck building block is introduced. Instead of a stack of two 3×33 \times 3 convolutions, the bottleneck design uses a stack of three layers:

  1. A 1×11 \times 1 convolution to reduce dimensions.
  2. A 3×33 \times 3 convolution (the bottleneck) operating on lower dimensions.
  3. A 1×11 \times 1 convolution to restore (increase) dimensions.
Sources

S4.SS1.p11.6

Deeper Bottleneck Architectures. Next we describe our deeper nets for ImageNet. Because of concerns on the training time that we can afford, we modify the building block as a bottleneck design444Deeper non-bottleneck ResNets (e.g., Fig. 5 left) also gain accuracy from increased depth (as shown on CIFAR-10), but are not as economical as the bottleneck ResNets. So the usage of bottleneck designs is mainly due to practical considerations. We further note that the degradation problem of plain nets is also witnessed for the bottleneck designs.. For each residual function ℱℱ\mathcal{F}, we use a stack of 3 layers instead of 2 (Fig. 5). The three layers are 1×\times1, 3×\times3, and 1×\times1 convolutions, where the 1×\times1 layers are responsible for reducing and then increasing (restoring) dimensions, leaving the 3×\times3 layer a bottleneck with smaller input/output dimensions. Fig. 5 shows an example, where both designs have similar time complexity.

S4.F5

Figure 5: A deeper residual function ℱℱ\mathcal{F} for ImageNet. Left: a building block (on 56×\times56 feature maps) as in Fig. 3 for ResNet-34. Right: a “bottleneck” building block for ResNet-50/101/152.
Deep dive

The parameter-free identity shortcut is crucial here. If the identity shortcut in the bottleneck block were replaced with a projection shortcut, the time complexity and model size would double because the shortcut connects two high-dimensional ends. Using identity shortcuts ensures the bottleneck design remains highly efficient.

Sources

S4.SS1.p12.1

The parameter-free identity shortcuts are particularly important for the bottleneck architectures. If the identity shortcut in Fig. 5 (right) is replaced with projection, one can show that the time complexity and model size are doubled, as the shortcut is connected to the two high-dimensional ends. So identity shortcuts lead to more efficient models for the bottleneck designs.