You are reading immutable version 28. The current guide may be newer.

Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Comprehensive Assessment

Test your understanding

Check your understanding

1. Why do recurrent neural network models inherently struggle with parallelization within training examples?
2. Explain how the sequential nature of recurrent neural networks limits batching across training examples when dealing with longer sequence lengths.
3. What is the primary reason for scaling the dot product of queries and keys by $1/\sqrt{d_k}$ in Scaled Dot-Product Attention?
4. Write the mathematical formula for Scaled Dot-Product Attention and explain why the variance of the dot product of a query $q$ and a key $k$ is $d_k$ if their components are independent random variables with mean 0 and variance 1.
5. What advantage does Multi-Head Attention provide over using a single attention head?
6. Describe the sequence of operations in Multi-Head Attention, starting from the linear projections of queries, keys, and values to the final output projection.
7. Under what condition are self-attention layers computationally faster than recurrent layers according to the paper?
8. Compare the per-layer complexity and the minimum number of sequential operations of a Self-Attention layer versus a Recurrent layer.
9. Why must the Transformer model inject positional encodings into the input embeddings?
10. Write the formula for the positional encoding at an even index $2i$ and explain how the positional encodings are combined with the input embeddings.
11. According to Table 2, what was the training cost (in FLOPs) of the Transformer (base model) on the English-to-German (EN-DE) translation task?
12. Discuss the performance and training efficiency of the Transformer (big) model on the WMT 2014 English-to-German translation task compared to previous state-of-the-art models.