You are reading immutable version 29. The current guide may be newer.

Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Scaled dot-product attention

Explore attention score scaling

Interactive study view

Explore attention score scaling

Illustrative teaching data

Move across fixed illustrative scenarios to see how score scaling changes concentration.

Peak attention weight0.7255227910591404 probabilityIllustrative teaching data
Validated scenario table
ScenarioAttention score scaleOutputProvenance
0.25×0.25 scale multiplier0.4087009733881603 probabilityIllustrative teaching data
0.5×0.5 scale multiplier0.5792585299413737 probabilityIllustrative teaching data
0.75×0.75 scale multiplier0.7255227910591404 probabilityIllustrative teaching data
1×1 scale multiplier0.8309526605439513 probabilityIllustrative teaching data
1.25×1.25 scale multiplier0.8990051408353833 probabilityIllustrative teaching data