You are reading immutable version 2. The current guide may be newer.

Study the paper

Deep Residual Learning for Image Recognition

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Mathematical Formulation and Shortcut Connections

Mathematical Formulation and Shortcut Connections

Mathematical Formulation and Shortcut Connections

Mathematical Formulation

Sources

S3.E1

𝐲=ℱ​(𝐱,{Wi})+𝐱.𝐲ℱ𝐱subscript𝑊𝑖𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}. (1)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}.

Formally, a residual block is defined by the following equation:

Sources

S3.E1

𝐲=ℱ​(𝐱,{Wi})+𝐱.𝐲ℱ𝐱subscript𝑊𝑖𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}. (1)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}.

y=F(x,{Wi})+x.\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}.

Sources

S3.E1

𝐲=ℱ​(𝐱,{Wi})+𝐱.𝐲ℱ𝐱subscript𝑊𝑖𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}. (1)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}.

Here, x\mathbf{x} and y\mathbf{y} are the input and output vectors of the layers considered, and the function F(x,{Wi})\mathcal{F}(\mathbf{x}, \{W_i\}) represents the residual mapping to be learned. This formulation introduces no extra parameters or computational complexity, allowing for fair comparisons with plain networks.

Sources

S3.SS2.p2.1

The shortcut connections in Eqn.(1) introduce neither extra parameter nor computation complexity. This is not only attractive in practice but also important in our comparisons between plain and residual networks. We can fairly compare plain/residual networks that simultaneously have the same number of parameters, depth, width, and computational cost (except for the negligible element-wise addition).

Projection Shortcuts

Sources

S3.E2

𝐲=ℱ​(𝐱,{Wi})+Ws​𝐱.𝐲ℱ𝐱subscript𝑊𝑖subscript𝑊𝑠𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}.

When input and output dimensions differ (for example, when changing channel depths), a projection shortcut WsW_s can be used to match dimensions:

Sources

S3.E2

𝐲=ℱ​(𝐱,{Wi})+Ws​𝐱.𝐲ℱ𝐱subscript𝑊𝑖subscript𝑊𝑠𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}.

y=F(x,{Wi})+Wsx.\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}.

Sources

S3.E2

𝐲=ℱ​(𝐱,{Wi})+Ws​𝐱.𝐲ℱ𝐱subscript𝑊𝑖subscript𝑊𝑠𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}.

We can investigate three design options for handling dimension changes:

  • (A) Zero-padding shortcuts are used for increasing dimensions, and all other shortcuts are parameter-free identity mappings.
  • (B) Projection shortcuts are used for increasing dimensions, and other shortcuts are identity.
  • (C) All shortcuts are projection shortcuts.
Sources

S4.SS1.p9.1

Identity vs. Projection Shortcuts. We have shown that parameter-free, identity shortcuts help with training. Next we investigate projection shortcuts (Eqn.(2)). In Table 3 we compare three options: (A) zero-padding shortcuts are used for increasing dimensions, and all shortcuts are parameter-free (the same as Table 2 and Fig. 4 right); (B) projection shortcuts are used for increasing dimensions, and other shortcuts are identity; and (C) all shortcuts are projections.