DeepStudy

Check your understanding

1. Which of the following best describes the 'degradation problem' observed in deep plain networks as depth increases?

Accuracy saturates and then degrades rapidly, accompanied by higher training error, which is not caused by overfitting.Accuracy degrades rapidly due to severe overfitting, leading to extremely low training error but high test error.The network fails to converge during the initial training epochs due to vanishing gradients.The model's parameter count exceeds hardware memory limits, causing training to halt.

2. Explain why the degradation problem in deep plain networks is not considered an overfitting issue, referencing the relationship between training error and network depth.

3. In the residual learning framework, if the desired underlying mapping is $\mathcal{H}(\mathbf{x})$, how is the residual mapping $\mathcal{F}(\mathbf{x})$ defined?

$\mathcal{F}(\mathbf{x}) := \mathcal{H}(\mathbf{x}) - \mathbf{x}$$\mathcal{F}(\mathbf{x}) := \mathcal{H}(\mathbf{x}) + \mathbf{x}$$\mathcal{F}(\mathbf{x}) := \mathbf{x} \cdot \mathcal{H}(\mathbf{x})$$\mathcal{F}(\mathbf{x}) := \mathcal{H}(\mathbf{x}) / \mathbf{x}$

4. Explain why it is hypothesized to be easier to optimize the residual mapping $\mathcal{F}(\mathbf{x}) = \mathcal{H}(\mathbf{x}) - \mathbf{x}$ than the original unreferenced mapping $\mathcal{H}(\mathbf{x})$, particularly in the extreme case where an identity mapping is optimal.

5. When the dimensions of the input and output of a residual block differ, which of the following options can be used to match the dimensions?

Using identity shortcuts with extra zero entries padded, or using projection shortcuts via $1 \times 1$ convolutions.Only projection shortcuts via $3 \times 3$ convolutions are allowed.Only zero-padding is allowed, as projection shortcuts always double the parameter count.The dimensions must always be identical; shortcuts cannot go across different feature map sizes.

6. Compare Option A (zero-padding shortcuts) and Option B (projection shortcuts) for handling dimension increases in residual connections. What is the primary trade-off regarding parameters?

7. What is the sequence of layers in a 'bottleneck' building block for deeper ResNets (such as ResNet-50/101/152)?

A stack of $1 \times 1$, $3 \times 3$, and $1 \times 1$ convolutions.A stack of $3 \times 3$, $3 \times 3$, and $3 \times 3$ convolutions.A stack of $5 \times 5$ and $3 \times 3$ convolutions.A single $7 \times 7$ convolution followed by a pooling layer.

8. Why are parameter-free identity shortcuts particularly important for the bottleneck architectures compared to replacing them with projection shortcuts?

9. What does the empirical analysis of the standard deviations of layer responses reveal about ResNets compared to plain networks?

ResNets generally have smaller response magnitudes than their plain counterparts, supporting the idea that residual functions are closer to zero.ResNets have significantly larger response magnitudes, indicating stronger signal propagation.The response magnitudes are identical, showing that residual connections do not affect layer outputs.Deeper ResNets have larger response magnitudes than shallower ResNets.

10. Based on the empirical analysis of layer responses, how does the magnitude of responses change as the depth of the ResNet increases (e.g., comparing ResNet-20, 56, and 110)?

11. Why does the 1202-layer ResNet perform worse on the CIFAR-10 test set than the 110-layer ResNet, despite having similar training error?

It suffers from overfitting because the model is unnecessarily large (19.4M parameters) for the small dataset and was trained without strong regularization like dropout.It suffers from severe optimization difficulties, preventing the training error from converging.The learning rate was too high, causing the gradients to explode during training.The 1202-layer network has fewer parameters than the 110-layer network, leading to underfitting.

12. Describe the performance of the 1202-layer ResNet on CIFAR-10 in terms of training error and test error, and explain how the authors propose to address its performance gap in future work.

Deep Residual Learning for Image Recognition

Test your understanding

Check your understanding