Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Overview of Scaling Laws in Language Models

At a glance

Scaling Laws in Language Models

Sources

S1.F1

Figure 1: Language modeling performance improves smoothly as we increase the model size, datasetset size, and amount of compute111Here we display predicted compute when using a sufficiently small batch size. See Figure 13 for comparison to the purely empirical data. used for training. For optimal performance all three factors must be scaled up in tandem. Empirical performance has a power-law relationship with each individual factor when not bottlenecked by the other two.

Language modeling performance improves smoothly as we scale up three key factors: model size $N$ , dataset size $D$ , and the total amount of training compute $C$ . When performance is not bottlenecked by any single factor, the cross-entropy loss follows a power-law relationship with each individual factor across several orders of magnitude.

Sources

S1.F1

Figure 1: Language modeling performance improves smoothly as we increase the model size, datasetset size, and amount of compute111Here we display predicted compute when using a sufficiently small batch size. See Figure 13 for comparison to the purely empirical data. used for training. For optimal performance all three factors must be scaled up in tandem. Empirical performance has a power-law relationship with each individual factor when not bottlenecked by the other two.

S1.SS1.SSS0.Px2.p1.1

Performance has a power-law relationship with each of the three scale factors N,D,C𝑁𝐷𝐶N,D,C when not bottlenecked by the other two, with trends spanning more than six orders of magnitude (see Figure 1). We observe no signs of deviation from these trends on the upper end, though performance must flatten out eventually before reaching zero loss. (Section 3)