You are reading immutable version 5. The current guide may be newer.

Study the paper

Deep Residual Learning for Image Recognition

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Mathematics of Identity Shortcuts and Dimension Matching

Mathematics of Identity Shortcuts and Dimension Matching

Mathematics of Identity Shortcuts and Dimension Matching

Mathematical Formulations of Shortcuts

Sources

S3.E1

𝐲=ℱ​(𝐱,{Wi})+𝐱.𝐲ℱ𝐱subscript𝑊𝑖𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}. (1)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}

S3.E2

𝐲=ℱ​(𝐱,{Wi})+Ws​𝐱.𝐲ℱ𝐱subscript𝑊𝑖subscript𝑊𝑠𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}

When the input and output dimensions of the residual block are identical, we define the residual block using an identity shortcut connection:

Sources

S3.E1

𝐲=ℱ​(𝐱,{Wi})+𝐱.𝐲ℱ𝐱subscript𝑊𝑖𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}. (1)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}

y=F(x,{Wi})+x\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}

Sources

S3.E1

𝐲=ℱ​(𝐱,{Wi})+𝐱.𝐲ℱ𝐱subscript𝑊𝑖𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}. (1)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+\mathbf{x}

This formulation introduces no extra parameters or computational complexity. However, when dimensions change (e.g., when downsampling), we can perform a linear projection WsW_s on the shortcut connection to match dimensions:

Sources

S3.E2

𝐲=ℱ​(𝐱,{Wi})+Ws​𝐱.𝐲ℱ𝐱subscript𝑊𝑖subscript𝑊𝑠𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}

y=F(x,{Wi})+Wsx\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}

Sources

S3.E2

𝐲=ℱ​(𝐱,{Wi})+Ws​𝐱.𝐲ℱ𝐱subscript𝑊𝑖subscript𝑊𝑠𝐱\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}. (2)
\mathbf{y}=\mathcal{F}(\mathbf{x},\{W_{i}\})+W_{s}\mathbf{x}