Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Integrated assessment

Test your understanding

1. What does the supplied architecture replace with attention?

RecurrenceValue vectorsSoftmax

2. State the architectural motivation established by the supplied introduction excerpt.

3. Which two major structural parts are named in the excerpt?

Encoder and decoderQuery and keyTraining and test sets

4. How would you begin tracing a component through the architecture?

5. Which matrices form the score product before scaling?

Q and the transpose of KQ and VK and V

6. Describe the roles of queries, keys, and values in the attention equation.

7. What divides the query-key score in scaled dot-product attention?

The square root of the key dimensionThe number of value vectorsThe number of decoder layers

8. Write the scaling divisor and explain what its symbol denotes.

9. In an illustrative scalar score of 2 with d_k = 4, what is the scaled score before softmax?

124

10. For an illustrative unscaled score s and key dimension d_k, state the score sent to softmax.