Study the paper
Scaling Laws for Neural Language Models
Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.
Optimal Allocation of Compute Budgets
Optimal Allocation of Compute Budgets
Optimal Allocation of Compute Budgets
Language modeling performance improves smoothly as we increase the model size, dataset size, and amount of compute used for training. Empirical performance has a power-law relationship with each individual factor when not bottlenecked by the other two.
Sources
S1.F1
Figure 1: Language modeling performance improves smoothly as we increase the model size, datasetset size, and amount of compute111Here we display predicted compute when using a sufficiently small batch size. See Figure 13 for comparison to the purely empirical data. used for training. For optimal performance all three factors must be scaled up in tandem. Empirical performance has a power-law relationship with each individual factor when not bottlenecked by the other two.
Each value of the compute budget has an associated optimal model size . Optimal model size grows very rapidly with , increasing by 5x for each 10x increase in compute. The number of data examples processed makes up the remainder of the increase, growing relatively modestly by only 2x.
Sources
S6.F14
Figure 14: Left: Each value of the compute budget Cminsubscript𝐶minC_{\rm min} has an associated optimal model size N𝑁N. Optimal model size grows very rapidly with Cminsubscript𝐶minC_{\rm min}, increasing by 5x for each 10x increase in compute. The number of data examples processed makes up the remainder of the increase, growing relatively modestly by only 2x. Right: The batch-adjusted number of optimization steps also grows very slowly, if at all, meaning that most of the growth in data examples processed can be used for increased batch sizes.
Far beyond the model sizes studied empirically, we find a contradiction between our equations for and due to the slow growth of data needed for compute-efficient training. The intersection marks the point before which we expect our predictions to break down.
Sources
S6.F15
Figure 15: Far beyond the model sizes we study empirically, we find a contradiction between our equations for L(Cmin)𝐿subscript𝐶minL(C_{\rm min}) and L(D)𝐿𝐷L(D) due to the slow growth of data needed for compute-efficient training. The intersection marks the point before which we expect our predictions to break down. The location of this point is highly sensitive to the precise exponents from our power-law fits.
| Measure | Value |
|---|
Sources
S1.F1
Figure 1: Language modeling performance improves smoothly as we increase the model size, datasetset size, and amount of compute111Here we display predicted compute when using a sufficiently small batch size. See Figure 13 for comparison to the purely empirical data. used for training. For optimal performance all three factors must be scaled up in tandem. Empirical performance has a power-law relationship with each individual factor when not bottlenecked by the other two.
S6.F14
Figure 14: Left: Each value of the compute budget Cminsubscript𝐶minC_{\rm min} has an associated optimal model size N𝑁N. Optimal model size grows very rapidly with Cminsubscript𝐶minC_{\rm min}, increasing by 5x for each 10x increase in compute. The number of data examples processed makes up the remainder of the increase, growing relatively modestly by only 2x. Right: The batch-adjusted number of optimization steps also grows very slowly, if at all, meaning that most of the growth in data examples processed can be used for increased batch sizes.
S6.F15
Figure 15: Far beyond the model sizes we study empirically, we find a contradiction between our equations for L(Cmin)𝐿subscript𝐶minL(C_{\rm min}) and L(D)𝐿𝐷L(D) due to the slow growth of data needed for compute-efficient training. The intersection marks the point before which we expect our predictions to break down. The location of this point is highly sensitive to the precise exponents from our power-law fits.