DeepStudy

Check your understanding

1. What is the "degradation problem" observed in deep neural networks as depth increases, and how is it distinguished from overfitting?

As depth increases, training accuracy saturates and then degrades rapidly, leading to higher training error, which is not caused by overfitting.As depth increases, training error decreases to near zero while test error increases rapidly, which is a classic sign of overfitting.As depth increases, the network fails to converge at all due to vanishing gradients, resulting in random-guess accuracy from the start.As depth increases, the model size becomes too large for GPU memory, causing training to halt.

2. Explain the "solution by construction" argument that demonstrates why the degradation problem is an optimization issue rather than a capacity issue.

3. If H(x) is the desired underlying mapping, how is the residual mapping F(x) formulated, and what is the extreme case hypothesis regarding its optimization?

F(x) = H(x) + x; if identity mapping is optimal, it is easier to push F(x) to one.F(x) = H(x) - x; if identity mapping is optimal, it is easier to push the residual F(x) to zero than to fit an identity mapping with stacked nonlinear layers.F(x) = x - H(x); if identity mapping is optimal, it is easier to scale F(x) to infinity.F(x) = H(x) * x; if identity mapping is optimal, it is easier to set F(x) to identity.

4. Mathematically define the relationship between the desired underlying mapping H(x) and the residual mapping F(x) to be fit by stacked layers, and explain why this formulation is hypothesized to be easier to learn.

5. What are the primary advantages of using parameter-free identity shortcut connections as formulated in y = F(x, {W_i}) + x?

They double the number of parameters to allow more complex representations.They add neither extra parameters nor computational complexity, allowing fair comparisons with plain networks and end-to-end training via SGD.They require modifying the optimization solvers to handle non-differentiable operations.They reduce the depth of the network by skipping half of the layers during backpropagation.

6. Explain how the formulation y = F(x, {W_i}) + x enables a fair comparison between plain and residual networks.

7. In comparing options A, B, and C for shortcuts when dimensions increase, what did the authors observe about the necessity of projection shortcuts?

Option C (all projection shortcuts) is vastly superior, proving that projection shortcuts are essential to solve the degradation problem.Option A (zero-padding for increasing dimensions, all others identity) performed poorly, showing that zero-padding cannot be used.The small differences among A, B, and C indicate that projection shortcuts are not essential for addressing the degradation problem.Projection shortcuts are only useful when the network depth is less than 20 layers.

8. Write down the equation for a projection shortcut and explain why option B (projection shortcuts only for increasing dimensions, others identity) is slightly better than option A (zero-padding for increasing dimensions).

9. What is the structure of the 3-layer "bottleneck" building block used in deeper ResNets (such as ResNet-50/101/152), and what is its primary purpose?

A stack of three 3x3 convolutions designed to maximize spatial receptive fields.A stack of 1x1, 3x3, and 1x1 convolutions, where the 1x1 layers reduce and then restore dimensions, leaving the 3x3 layer a bottleneck to control computational complexity.A stack of 5x5, 3x3, and 1x1 convolutions designed to eliminate the need for identity shortcuts.A single 3x3 convolution followed by two fully connected layers to reduce model size.

10. Explain why parameter-free identity shortcuts are particularly important for the bottleneck architectures compared to projection shortcuts.

11. What does the empirical analysis of the standard deviations of layer responses on CIFAR-10 reveal about ResNets compared to plain networks?

ResNets have generally larger responses, showing that residual functions are highly active.ResNets have generally smaller responses, supporting the motivation that residual functions are generally closer to zero than non-residual functions.Deeper ResNets have larger response magnitudes than shallower ResNets.There is no difference in response magnitudes between ResNets and plain networks.

12. Based on the empirical analysis of layer responses, how does the magnitude of responses change as the depth of the ResNet increases (e.g., comparing ResNet-20, 56, and 110), and what does this imply about individual layers?

13. When exploring an aggressively deep model of 1202 layers on CIFAR-10, what optimization and generalization behaviors were observed?

The model showed severe optimization difficulties and failed to achieve a training error below 10%.The model showed no optimization difficulty (achieving training error < 0.1%), but its test error was worse than the 110-layer network due to overfitting.The model achieved both lower training and lower test error than all shallower networks without any regularization.The model was too large to be trained and diverged within the first few epochs.

14. Why did the 1202-layer ResNet perform worse on the test set than the 110-layer ResNet despite having extremely low training error, and what solutions do the authors suggest?

Deep Residual Learning for Image Recognition

Test your understanding

Check your understanding