Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Empirical Results and Performance

Empirical Evaluation and Efficiency

Sources

S6.T2

Table 2: The Transformer achieves better BLEU scores than previous state-of-the-art models on the English-to-German and English-to-French newstest2014 tests at a fraction of the training cost. Model BLEU Training Cost (FLOPs) EN-DE EN-FR EN-DE EN-FR ByteNet [18] 23.75 Deep-Att + PosUnk [39] 39.2 1.0⋅10201.0\cdot 10^{20} GNMT + RL [38] 24.6 39.92 2.3⋅10192.3\cdot 10^{19} 1.4⋅10201.4\cdot 10^{20} ConvS2S [9] 25.16 40.46 9.6⋅10189.6\cdot 10^{18} 1.5⋅10201.5\cdot 10^{20} MoE [32] 26.03 40.56 2.0⋅10192.0\cdot 10^{19} 1.2⋅10201.2\cdot 10^{20} Deep-Att + PosUnk Ensemble [39] 40.4 8.0⋅10208.0\cdot 10^{20} GNMT + RL Ensemble [38] 26.30 41.16 1.8⋅10201.8\cdot 10^{20} 1.1⋅10211.1\cdot 10^{21} ConvS2S Ensemble [9] 26.36 41.29 7.7⋅10197.7\cdot 10^{19} 1.2⋅10211.2\cdot 10^{21} Transformer (base model) 27.3 38.1 3.3⋅𝟏𝟎𝟏𝟖3.3\cdot 10^{18} Transformer (big) 28.4 41.8 2.3⋅10192.3\cdot 10^{19}

The Transformer achieves state-of-the-art results on translation tasks while requiring significantly less training cost. On the WMT 2014 English-to-German translation task, the big Transformer model establishes a new state-of-the-art BLEU score of 28.4, outperforming previous ensemble models by more than 2.0 BLEU. This was achieved at a fraction of the training cost of competitive models.

Sources

S6.SS1.p1.4

On the WMT 2014 English-to-German translation task, the big transformer model (Transformer (big) in Table 2) outperforms the best previously reported models (including ensembles) by more than 2.02.0 BLEU, establishing a new state-of-the-art BLEU score of 28.428.4. The configuration of this model is listed in the bottom line of Table 3. Training took 3.53.5 days on 88 P100 GPUs. Even our base model surpasses all previously published models and ensembles, at a fraction of the training cost of any of the competitive models.

Deep dive

In addition to machine translation, the Transformer generalizes exceptionally well to English constituency parsing, achieving high F1 scores in both discriminative and semi-supervised settings (e.g., reaching 92.7 F1 in the semi-supervised setup).

Sources

S6.T4

Table 4: The Transformer generalizes well to English constituency parsing (Results are on Section 23 of WSJ) Parser Training WSJ 23 F1 Vinyals & Kaiser el al. (2014) [37] WSJ only, discriminative 88.3 Petrov et al. (2006) [29] WSJ only, discriminative 90.4 Zhu et al. (2013) [40] WSJ only, discriminative 90.4 Dyer et al. (2016) [8] WSJ only, discriminative 91.7 Transformer (4 layers) WSJ only, discriminative 91.3 Zhu et al. (2013) [40] semi-supervised 91.3 Huang & Harper (2009) [14] semi-supervised 91.3 McClosky et al. (2006) [26] semi-supervised 92.1 Vinyals & Kaiser el al. (2014) [37] semi-supervised 92.1 Transformer (4 layers) semi-supervised 92.7 Luong et al. (2015) [23] multi-task 93.0 Dyer et al. (2016) [8] generative 93.3