Deep Residual Learning for Image Recognition

This equation defines the fundamental identity shortcut connection in Deep Residual Learning. Instead of forcing stacked layers to directly fit a desired underlying mapping, the network is reformulated to let these layers fit a residual mapping $\mathcal{F}(\mathbf{x}, \{W_i\})$. The original input $\mathbf{x}$ is added directly to the output of the residual function via a shortcut connection.

The identity shortcut connection adds no extra parameters or computational complexity to the network. It allows gradients to flow directly back through the shortcut connection, mitigating the vanishing gradient problem in extremely deep networks.

Illustrative Toy Calculation

Let us consider a simple 1D case where:

• Input $\mathbf{x} = [2.0]$
• Weights $\{W_i\}$ represent a single scaling factor $W = [0.5]$
• The residual function is a simple linear transformation: $\mathcal{F}(\mathbf{x}, \{W_i\}) = W \cdot \mathbf{x} = 0.5 \times 2.0 = 1.0$

Applying the identity shortcut connection: $\mathbf{y} = \mathcal{F}(\mathbf{x}, \{W_i\}) + \mathbf{x} = 1.0 + 2.0 = 3.0$