DeepStudy

Check your understanding

1. Which of the following statements accurately describes the 'degradation problem' in deep neural networks as observed by the authors?

As depth increases, training error decreases to near zero while test error increases rapidly due to overfitting.As depth increases, training error increases, indicating that deeper models are harder for current solvers to optimize.As depth increases, both training and test error decrease exponentially without saturation.As depth increases, optimization becomes easier but generalization error increases due to parameter explosion.

2. Explain why the degradation problem is distinct from overfitting, referencing the relationship between training error and network depth.

3. In the residual learning framework, if the optimal mapping $\mathcal{H}(\mathbf{x})$ is an identity mapping, how does the reformulation $\mathcal{F}(\mathbf{x}) = \mathcal{H}(\mathbf{x}) - \mathbf{x}$ make optimization easier for the solver?

4. If the desired underlying mapping is $\mathcal{H}(\mathbf{x})$, which of the following represents the recast formulation learned by the residual framework?

$\mathcal{F}(\mathbf{x}) = \mathcal{H}(\mathbf{x}) \cdot \mathbf{x}$$\mathcal{F}(\mathbf{x}) + \mathbf{x}$ where $\mathcal{F}(\mathbf{x}) = \mathcal{H}(\mathbf{x}) - \mathbf{x}$$\mathcal{F}(\mathbf{x}) - \mathbf{x}$ where $\mathcal{F}(\mathbf{x}) = \mathcal{H}(\mathbf{x}) + \mathbf{x}$$\mathcal{F}(\mathbf{x}) = \mathbf{x} - \mathcal{H}(\mathbf{x})$

5. What is a key advantage of using identity shortcut connections in residual networks?

They double the number of parameters to increase representation capacity.They introduce zero extra parameters and no additional computational complexity.They replace backpropagation with a simpler feedforward optimization solver.They require modifying standard SGD solvers to handle non-differentiable operations.

6. Explain how identity shortcut connections are implemented mathematically and why they do not increase the parameter count of the network.

7. Compare Option A (zero-padding) and Option B (projection shortcuts) for matching dimensions when they go across feature maps of different sizes. Why does Option B perform slightly better than Option A?

8. Based on the authors' comparison of shortcut options (A, B, and C), which of the following is a key reason why they chose NOT to use Option C (all projection shortcuts) in the rest of the paper?

Option C resulted in significantly worse accuracy than the plain counterpart.Option C is avoided to reduce memory/time complexity and model sizes, as projection shortcuts are not essential for addressing degradation.Option C cannot be trained end-to-end using standard SGD solvers.Option C does not support bottleneck architectures.

9. Describe the structure of the 3-layer bottleneck block and explain why using identity shortcuts (instead of projection shortcuts) is particularly critical for this design's efficiency.

10. In the deeper bottleneck architecture, what are the specific roles of the 1x1 convolutions?

They perform spatial pooling to reduce the resolution of the feature maps.They are responsible for reducing and then restoring (increasing) dimensions, leaving the 3x3 layer with smaller dimensions.They act as the primary feature extraction layers with large receptive fields.They replace the element-wise addition at the end of the residual block.

11. What does the empirical analysis of layer responses (standard deviations) reveal about ResNets compared to plain networks?

12. How does the magnitude of layer responses change as the depth of a ResNet increases (e.g., comparing ResNet-20 to ResNet-110)?

Deeper ResNets have larger response magnitudes, indicating stronger feature extraction.Deeper ResNets have smaller response magnitudes, meaning individual layers tend to modify the signal less.The response magnitude remains completely constant regardless of depth.The response magnitude fluctuates randomly without any observable trend.

Deep Residual Learning for Image Recognition

Test your understanding

Check your understanding