DeepStudy

Check your understanding

1. In the Transformer's encoder-decoder structure, how does the decoder generate the output sequence $(y_1, ..., y_m)$ given the continuous representations $\mathbf{z}$ produced by the encoder?

2. Which of the following best describes the mapping performed by the Transformer's encoder?

It maps an input sequence of symbol representations to a sequence of continuous representations.It maps continuous representations directly to discrete output probabilities in one step.It maps an input sequence to a single fixed-size context vector.It maps output symbols back to input symbol representations.

3. Write down the mathematical formula for Scaled Dot-Product Attention as a function of queries $Q$, keys $K$, and values $V$, where $d_k$ represents the dimension of the keys.

4. What is the primary purpose of the scaling factor $\frac{1}{\sqrt{d_k}}$ in Scaled Dot-Product Attention?

To prevent the dot products from growing extremely large in magnitude, which would push the softmax function into regions with very small gradients.To reduce the computational complexity of the matrix multiplication from $O(n^2 \cdot d)$ to $O(n \cdot d)$.To project the queries and keys into a lower-dimensional subspace before computing attention.To ensure that the attention weights sum to exactly 1 without needing a softmax function.

5. According to the authors, what limitation of a single attention head does multi-head attention overcome?

6. Which of the following is a key benefit of using multi-head attention instead of a single attention head?

It allows the model to jointly attend to information from different representation subspaces at different positions.It reduces the total parameter count of the attention layer.It eliminates the need for any feed-forward layers in the network.It guarantees that the attention matrix is always diagonal.

7. Write down the mathematical formula for the positional encoding at an even dimension index $2i$ for a given position $pos$ and model dimension $d_{\text{model}}$.

8. Why are positional encodings necessary in the Transformer architecture?

Because the model contains no recurrence and no convolution, so positional encodings are needed to inject information about the sequence order.To scale the input embeddings so that their values do not exceed the numerical range of the softmax function.To project the input tokens into a continuous vector space of dimension $d_{\text{model}}$.To allow the decoder to attend to the encoder outputs in an auto-regressive manner.

9. Compare Self-Attention and Recurrent layers in terms of their complexity per layer and maximum path length as a function of sequence length $n$ and representation dimension $d$.

10. According to Table 1, what is the minimum number of sequential operations required by a Self-Attention layer compared to a Recurrent layer?

Self-Attention: $O(1)$; Recurrent: $O(n)$Self-Attention: $O(n)$; Recurrent: $O(1)$Self-Attention: $O(n^2)$; Recurrent: $O(n \cdot d^2)$Self-Attention: $O(1)$; Recurrent: $O(\log(n))$

11. Based on Table 2, what BLEU scores did the 'Transformer (big)' model achieve on the English-to-German (EN-DE) and English-to-French (EN-FR) newstest2014 tests?

12. According to Table 2, how does the training cost (in FLOPs) of the 'Transformer (base model)' compare to the 'GNMT + RL' model on the English-to-German (EN-DE) task?

The Transformer (base model) training cost ($3.3 \cdot 10^{18}$ FLOPs) is lower than that of GNMT + RL ($2.3 \cdot 10^{19}$ FLOPs).The Transformer (base model) training cost is higher than that of GNMT + RL.The training costs for both models are identical.The Transformer (base model) training cost is not reported in Table 2.

Attention Is All You Need

Test your understanding

Check your understanding