Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Scaled Dot-Product Attention

Illustrative teaching scenarios: attention scaling (not paper-reported)

Interactive study view

Illustrative teaching scenarios: attention scaling (not paper-reported)

Illustrative teaching data

Compare fixed illustrative teaching scenarios; these are not paper-reported results.

Illustrative teaching scenarios: attention scaling (not paper-reported) data
labellabelprobabilitylabelProvenance
0.5×Peak attention weight0.5792585299413737probabilityIllustrative teaching data
0.25×Peak attention weight0.4087009733881603probabilityIllustrative teaching data
0.75×Peak attention weight0.7255227910591404probabilityIllustrative teaching data
1.25×Peak attention weight0.8990051408353833probabilityIllustrative teaching data
1×Peak attention weight0.8309526605439513probabilityIllustrative teaching data