You are reading immutable version 40. The current guide may be newer.

Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Comprehensive Assessment

Test your understanding

Check your understanding

1. Why do recurrent models inherently struggle with parallelization within training examples?
2. Explain how the Transformer architecture addresses the sequential computation constraint of recurrent networks.
3. What is the primary reason for scaling the dot products by $1/\sqrt{d_k}$ in the attention mechanism?
4. Assuming that the components of $q$ and $k$ are independent random variables with mean 0 and variance 1, derive the mean and variance of their dot product $q \cdot k$, and explain how this justifies the scaling factor.
5. What limitation of a single attention head does Multi-Head Attention overcome?
6. Describe the sequence of operations in Multi-Head Attention, from the initial linear projections to the final output projection.
7. Why are positional encodings necessary in the Transformer model?
8. Write down the formulas for the sinusoidal positional encodings used in the Transformer and explain how their dimensions allow them to be combined with the input embeddings.
9. Under what condition is a self-attention layer computationally faster than a recurrent layer?
10. Compare the maximum path lengths of self-attention, recurrent, and convolutional layers, and explain how these path lengths affect the learning of long-range dependencies.
11. According to Table 2, how does the training cost (in FLOPs) of the Transformer (base model) compare to other competitive models on the English-to-German (EN-DE) translation task?
12. Discuss the performance of the Transformer on English constituency parsing tasks, specifically addressing how it performs relative to task-specific models despite the lack of task-specific tuning.