DeepStudy

Check your understanding

1. Why does the sequential nature of recurrent neural networks (RNNs) limit parallelization during training?

Because hidden states must be computed sequentially, as each state depends on the previous state, preventing parallel computation within a sequence.Because recurrent networks require extremely large batch sizes that exceed GPU memory limits.Because recurrent networks cannot use factorization tricks or conditional computation.Because recurrent networks are mathematically incompatible with backpropagation.

2. Explain how the maximum path length between any two input and output positions in a self-attention layer compares to a recurrent layer, and why this difference affects the model's ability to learn long-range dependencies.

3. What is the primary reason for scaling the dot products of queries and keys by $1/\sqrt{d_k}$ in the attention mechanism?

To reduce the computational complexity of the dot product from quadratic to linear.To prevent the dot products from growing too large and pushing the softmax function into regions with extremely small gradients.To ensure that the attention matrix is always symmetric and positive semi-definite.To align the dimensions of the queries and keys with the value vectors.

4. Assume that the components of $q$ and $k$ are independent random variables with mean 0 and variance 1. Show mathematically what the mean and variance of their dot product $q \cdot k$ are, and explain how this relates to the necessity of the scaling factor.

5. How does Multi-Head Attention prevent the averaging effect that occurs with a single attention head?

By performing attention over multiple independent sequences simultaneously.By projecting queries, keys, and values into multiple different representation subspaces, allowing the model to attend to different positions jointly.By applying a dropout layer directly to the attention weights of each head.By replacing the softmax function with a simple linear average.

6. Write down the mathematical formulation of Multi-Head Attention and explain the role of the projection matrices $W_i^Q$, $W_i^K$, $W_i^V$, and $W^O$.

7. Why must positional encodings be added to the input embeddings in the Transformer architecture?

To increase the dimensionality of the input embeddings so they can be processed by multi-head attention.To inject information about the relative or absolute position of the tokens, since the model contains no recurrence or convolution.To normalize the input embeddings and prevent gradient explosion.To allow the model to dynamically adjust the learning rate during training.

8. Explain why the positional encodings have the same dimension $d_{\text{model}}$ as the input embeddings, and write down the formulas for the even and odd dimensions of the sinusoidal positional encodings.

9. According to Table 2, how does the training cost (in FLOPs) of the Transformer (base model) compare to the GNMT + RL model on the English-to-German (EN-DE) translation task?

The Transformer (base model) has a higher training cost than GNMT + RL.The Transformer (base model) has a lower training cost ($3.3 \cdot 10^{18}$ FLOPs) compared to GNMT + RL ($2.3 \cdot 10^{19}$ FLOPs).Both models have identical training costs of $1.5 \cdot 10^{20}$ FLOPs.The Transformer (base model) does not report training cost in FLOPs.

10. Based on the empirical results in Table 4, evaluate the performance (WSJ 23 F1 score) of the Transformer (4 layers) under the semi-supervised setup compared to other semi-supervised models.

Attention Is All You Need

Test your understanding

Check your understanding