You are reading immutable version 40. The current guide may be newer.

Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Scaled Dot-Product Attention

Illustrative teaching scenarios: attention concentration (not paper-reported)

Interactive study view

Illustrative teaching scenarios: attention concentration (not paper-reported)

Illustrative teaching data

Trace fixed illustrative teaching scenarios; these are not paper-reported results.

Peak attention weight, point 1: 0.25, 0.4087009733881603Peak attention weight, point 2: 0.5, 0.5792585299413737Peak attention weight, point 3: 0.75, 0.7255227910591404Peak attention weight, point 4: 1, 0.8309526605439513Peak attention weight, point 5: 1.25, 0.8990051408353833

Attention score scale (scale multiplier)

Peak attention weight (probability)

  • Peak attention weight
Illustrative teaching scenarios: attention concentration (not paper-reported) data
SeriesAttention score scalePeak attention weightProvenance
Peak attention weight0.250.4087009733881603Illustrative teaching data
Peak attention weight0.50.5792585299413737Illustrative teaching data
Peak attention weight0.750.7255227910591404Illustrative teaching data
Peak attention weight10.8309526605439513Illustrative teaching data
Peak attention weight1.250.8990051408353833Illustrative teaching data