Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Overview of Scaling Laws for Language Models

At a glance

Scaling Laws for Language Models

Sources

S1.F1

Figure 1: Language modeling performance improves smoothly as we increase the model size, datasetset size, and amount of compute111Here we display predicted compute when using a sufficiently small batch size. See Figure 13 for comparison to the purely empirical data. used for training. For optimal performance all three factors must be scaled up in tandem. Empirical performance has a power-law relationship with each individual factor when not bottlenecked by the other two.

Language modeling performance improves smoothly and predictably as we scale up three primary factors: model size $N$ , dataset size $D$ , and the total compute budget $C$ . When performance is not bottlenecked by the other two factors, the cross-entropy loss follows a power-law relationship with each individual factor, spanning over six orders of magnitude.

Sources

S1.F1

Figure 1: Language modeling performance improves smoothly as we increase the model size, datasetset size, and amount of compute111Here we display predicted compute when using a sufficiently small batch size. See Figure 13 for comparison to the purely empirical data. used for training. For optimal performance all three factors must be scaled up in tandem. Empirical performance has a power-law relationship with each individual factor when not bottlenecked by the other two.

S1.SS1.SSS0.Px2.p1.1

Performance has a power-law relationship with each of the three scale factors N,D,C𝑁𝐷𝐶N,D,C when not bottlenecked by the other two, with trends spanning more than six orders of magnitude (see Figure 1). We observe no signs of deviation from these trends on the upper end, though performance must flatten out eventually before reaching zero loss. (Section 3)

To obtain clean, un-obscured scaling laws, it is crucial to define the model size $N$ by excluding embedding parameters (such as the vocabulary embedding matrix $n_{\rm vocab} d_{\rm model}$ and positional embeddings $n_{\rm ctx} d_{\rm model}$ ). Including these parameters obscures the empirical trends, suggesting that the embedding matrix can be made smaller without impacting overall performance.

Sources

S2.SS1.p2.4

where we have excluded biases and other sub-leading terms. Our models also have nvocabdmodelsubscript𝑛vocabsubscript𝑑modeln_{\rm vocab}d_{{\rm model}} parameters in an embedding matrix, and use nctxdmodelsubscript𝑛ctxsubscript𝑑modeln_{\rm ctx}d_{{\rm model}} parameters for positional embeddings, but we do not include these when discussing the ‘model size’ N𝑁N; we will see that this produces significantly cleaner scaling laws.

S3.SS2.p2.2

To observe these trends it is crucial to study performance as a function of N𝑁N; if we instead use the total parameter count (including the embedding parameters) the trend is somewhat obscured (see Figure 6). This suggests that the embedding matrix can be made smaller without impacting performance, as has been seen in recent work [LCG+19].