You are reading immutable version 38. The current guide may be newer.

Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Scaled dot-product attention

Illustrative teaching scenarios: attention scaling (not paper-reported)

Interactive study view

Illustrative teaching scenarios: attention scaling (not paper-reported)

Illustrative teaching data

Compare fixed illustrative teaching scenarios; these are not paper-reported results.

Illustrative teaching scenarios: attention scaling (not paper-reported) data
labellabelprobabilitylabelProvenance
0.5×Peak attention weight0.5792585299413737probabilityIllustrative teaching data
0.25×Peak attention weight0.4087009733881603probabilityIllustrative teaching data
0.75×Peak attention weight0.7255227910591404probabilityIllustrative teaching data
1.25×Peak attention weight0.8990051408353833probabilityIllustrative teaching data
1×Peak attention weight0.8309526605439513probabilityIllustrative teaching data