Study the paper

Attention Is All You Need

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Empirical Performance and Generalization

Empirical Performance on Translation and Parsing

On the WMT 2014 English-to-German translation task, the big Transformer model achieves a state-of-the-art BLEU score of 28.4, outperforming previous models and ensembles by more than 2.0 BLEU. The base model also outperforms previous models at a fraction of the training cost (requiring only $3.3 \cdot 10^{18}$ FLOPs compared to over $10^{19}$ or $10^{20}$ FLOPs for competitive architectures).

Sources

S6.T2

Table 2: The Transformer achieves better BLEU scores than previous state-of-the-art models on the English-to-German and English-to-French newstest2014 tests at a fraction of the training cost. Model BLEU Training Cost (FLOPs) EN-DE EN-FR EN-DE EN-FR ByteNet [18] 23.75 Deep-Att + PosUnk [39] 39.2 1.0⋅10201.0\cdot 10^{20} GNMT + RL [38] 24.6 39.92 2.3⋅10192.3\cdot 10^{19} 1.4⋅10201.4\cdot 10^{20} ConvS2S [9] 25.16 40.46 9.6⋅10189.6\cdot 10^{18} 1.5⋅10201.5\cdot 10^{20} MoE [32] 26.03 40.56 2.0⋅10192.0\cdot 10^{19} 1.2⋅10201.2\cdot 10^{20} Deep-Att + PosUnk Ensemble [39] 40.4 8.0⋅10208.0\cdot 10^{20} GNMT + RL Ensemble [38] 26.30 41.16 1.8⋅10201.8\cdot 10^{20} 1.1⋅10211.1\cdot 10^{21} ConvS2S Ensemble [9] 26.36 41.29 7.7⋅10197.7\cdot 10^{19} 1.2⋅10211.2\cdot 10^{21} Transformer (base model) 27.3 38.1 3.3⋅𝟏𝟎𝟏𝟖3.3\cdot 10^{18} Transformer (big) 28.4 41.8 2.3⋅10192.3\cdot 10^{19}

S6.SS1.p1.4

On the WMT 2014 English-to-German translation task, the big transformer model (Transformer (big) in Table 2) outperforms the best previously reported models (including ensembles) by more than 2.02.0 BLEU, establishing a new state-of-the-art BLEU score of 28.428.4. The configuration of this model is listed in the bottom line of Table 3. Training took 3.53.5 days on 88 P100 GPUs. Even our base model surpasses all previously published models and ensembles, at a fraction of the training cost of any of the competitive models.

To test its generalization capabilities, the Transformer was evaluated on English constituency parsing. Despite the lack of task-specific tuning, the semi-supervised 4-layer Transformer achieved an F1 score of 92.7 on Section 23 of the WSJ, performing surprisingly well against highly specialized models.

Sources

S6.T4

Table 4: The Transformer generalizes well to English constituency parsing (Results are on Section 23 of WSJ) Parser Training WSJ 23 F1 Vinyals & Kaiser el al. (2014) [37] WSJ only, discriminative 88.3 Petrov et al. (2006) [29] WSJ only, discriminative 90.4 Zhu et al. (2013) [40] WSJ only, discriminative 90.4 Dyer et al. (2016) [8] WSJ only, discriminative 91.7 Transformer (4 layers) WSJ only, discriminative 91.3 Zhu et al. (2013) [40] semi-supervised 91.3 Huang & Harper (2009) [14] semi-supervised 91.3 McClosky et al. (2006) [26] semi-supervised 92.1 Vinyals & Kaiser el al. (2014) [37] semi-supervised 92.1 Transformer (4 layers) semi-supervised 92.7 Luong et al. (2015) [23] multi-task 93.0 Dyer et al. (2016) [8] generative 93.3

S6.SS3.p4.1

Our results in Table 4 show that despite the lack of task-specific tuning our model performs surprisingly well, yielding better results than all previously reported models with the exception of the Recurrent Neural Network Grammar [8].