Study the paper
Scaling Laws for Neural Language Models
Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.
Overview of Scaling Laws for Language Models
Overview of Scaling Laws for Language Models
Overview of Scaling Laws for Language Models
Power-Law Scaling of Language Models
Sources
S1.F1
Figure 1: Language modeling performance improves smoothly as we increase the model size, datasetset size, and amount of compute111Here we display predicted compute when using a sufficiently small batch size. See Figure 13 for comparison to the purely empirical data. used for training. For optimal performance all three factors must be scaled up in tandem. Empirical performance has a power-law relationship with each individual factor when not bottlenecked by the other two.
S1.SS1.SSS0.Px2.p1.1
Performance has a power-law relationship with each of the three scale factors N,D,C𝑁𝐷𝐶N,D,C when not bottlenecked by the other two, with trends spanning more than six orders of magnitude (see Figure 1). We observe no signs of deviation from these trends on the upper end, though performance must flatten out eventually before reaching zero loss. (Section 3)
Language modeling performance, measured via cross-entropy loss, improves smoothly as we scale up three primary factors: model size , dataset size , and the total compute budget . When performance is not bottlenecked by the other two factors, the cross-entropy loss exhibits a power-law relationship with each individual scale factor. These trends span more than six orders of magnitude, showing no signs of deviation on the upper end, though they must eventually flatten out before reaching a loss of zero.
Sources
S1.F1
Figure 1: Language modeling performance improves smoothly as we increase the model size, datasetset size, and amount of compute111Here we display predicted compute when using a sufficiently small batch size. See Figure 13 for comparison to the purely empirical data. used for training. For optimal performance all three factors must be scaled up in tandem. Empirical performance has a power-law relationship with each individual factor when not bottlenecked by the other two.
S1.SS1.SSS0.Px2.p1.1
Performance has a power-law relationship with each of the three scale factors N,D,C𝑁𝐷𝐶N,D,C when not bottlenecked by the other two, with trends spanning more than six orders of magnitude (see Figure 1). We observe no signs of deviation from these trends on the upper end, though performance must flatten out eventually before reaching zero loss. (Section 3)
Defining Model Size and the Role of Embeddings
Sources
S2.SS1.p2.4
where we have excluded biases and other sub-leading terms. Our models also have nvocabdmodelsubscript𝑛vocabsubscript𝑑modeln_{\rm vocab}d_{{\rm model}} parameters in an embedding matrix, and use nctxdmodelsubscript𝑛ctxsubscript𝑑modeln_{\rm ctx}d_{{\rm model}} parameters for positional embeddings, but we do not include these when discussing the ‘model size’ N𝑁N; we will see that this produces significantly cleaner scaling laws.
S3.SS2.p2.2
To observe these trends it is crucial to study performance as a function of N𝑁N; if we instead use the total parameter count (including the embedding parameters) the trend is somewhat obscured (see Figure 6). This suggests that the embedding matrix can be made smaller without impacting performance, as has been seen in recent work [LCG+19].
To observe clean scaling laws, the parameter count is defined to exclude the embedding parameters. Specifically, the vocabulary embeddings () and positional embeddings () are excluded from . Including these parameters obscures the power-law trends because the embedding matrix can be made smaller or modified without significantly impacting language modeling performance.
Sources
S2.SS1.p2.4
where we have excluded biases and other sub-leading terms. Our models also have nvocabdmodelsubscript𝑛vocabsubscript𝑑modeln_{\rm vocab}d_{{\rm model}} parameters in an embedding matrix, and use nctxdmodelsubscript𝑛ctxsubscript𝑑modeln_{\rm ctx}d_{{\rm model}} parameters for positional embeddings, but we do not include these when discussing the ‘model size’ N𝑁N; we will see that this produces significantly cleaner scaling laws.
S3.SS2.p2.2
To observe these trends it is crucial to study performance as a function of N𝑁N; if we instead use the total parameter count (including the embedding parameters) the trend is somewhat obscured (see Figure 6). This suggests that the embedding matrix can be made smaller without impacting performance, as has been seen in recent work [LCG+19].