Study the paper

Deep Residual Learning for Image Recognition

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

14 activities

Lesson

At a glance

Introduction to Residual Learning

Lesson

Introduction to Residual Learning

The Degradation Problem in Deep Networks

Lesson

Mathematics of Identity Shortcuts and Dimension Matching

Mathematical Formulations of Shortcuts

Lesson

Residual Block with Identity Shortcut

Mathematics of Identity Shortcuts and Dimension Matching

This equation defines the fundamental formulation of a residual learning block. Instead of directly fitting an underlying mapping with a stack of layers, the network is designed to approximate a residual mapping mathcal F ( mathbf x , W i ) = mathbf y - mathbf x . The original mapping is recast into mathcal F ( mathbf x ) + mathbf x , which is realized by a feedforward neural network with shortcut connections.

Lesson

Residual Block with Projection Shortcut

Mathematics of Identity Shortcuts and Dimension Matching

This equation defines a residual block with a projection shortcut, which is used to match dimensions when the input and output dimensions of a residual block differ.

Visual

Residual Block with Projection Shortcut: step by step

Mathematics of Identity Shortcuts and Dimension Matching

Follow the existing bounded walkthrough in its intended sequence.

Visual

Residual Block with Identity Shortcut: step by step

Mathematics of Identity Shortcuts and Dimension Matching

Follow the existing bounded walkthrough in its intended sequence.

Lesson

Bottleneck Architectures and Layer Responses

The Bottleneck Building Block Design

Lesson

Standard Deviations of Layer Responses on CIFAR-10

Bottleneck Architectures and Layer Responses

An analysis of the standard deviations (std) of layer responses on CIFAR-10 reveals key insights into the behavior of residual learning. The responses are measured at the outputs of each 3 times3 layer, after Batch Normalization (BN) and before the nonlinearity.

Lesson