You are reading immutable version 4. The current guide may be newer.

Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Overfitting and the Infinite Data Limit

Optimal Allocation Scaling

Optimal Allocation Scaling

Source equation

N∝CαCmin/αN,B∝CαCmin/αB,S∝CαCmin/αS,D=B⋅SN\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad

This equation describes the optimal scaling of model parameters (NN), batch size (BB), and training steps (SS) as power-law functions of the total compute budget (CC). It also defines the total dataset size in tokens (DD) as the product of batch size and training steps.

Sources

S1.E7

N∝CαCmin/αN,B∝CαCmin/αB,S∝CαCmin/αS,D=B⋅Sformulae-sequenceproportional-to𝑁superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝑁formulae-sequenceproportional-to𝐵superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝐵formulae-sequenceproportional-to𝑆superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝑆𝐷⋅𝐵𝑆N\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad (1.7)
N\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad

N∝CαCmin/αN,B∝CαCmin/αB,S∝CαCmin/αS,D=B⋅SN\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad

Sources

S1.E7

N∝CαCmin/αN,B∝CαCmin/αB,S∝CαCmin/αS,D=B⋅Sformulae-sequenceproportional-to𝑁superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝑁formulae-sequenceproportional-to𝐵superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝐵formulae-sequenceproportional-to𝑆superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝑆𝐷⋅𝐵𝑆N\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad (1.7)
N\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad
Implementation detail

Illustrative Calculation

Let us calculate the optimal scaling exponents given the following empirical values:

  • αN=0.076\alpha_N = 0.076
  • αB=0.21\alpha_B = 0.21
  • αS=0.17\alpha_S = 0.17

First, we compute the minimum compute scaling exponent αCmin\alpha_C^{\rm min}:

\right)^{-1} \approx (13.158 + 4.762 + 5.882)^{-1} \approx 23.802^{-1} \approx 0.0420$$ Now, we compute the power-law exponents for $N$, $B$, and $S$: - For $N$: $\frac{\alpha_C^{\rm min}}{\alpha_N} \approx \frac{0.0420}{0.076} \approx 0.55$ - For $B$: $\frac{\alpha_C^{\rm min}}{\alpha_B} \approx \frac{0.0420}{0.21} \approx 0.20$ - For $S$: $\frac{\alpha_C^{\rm min}}{\alpha_S} \approx \frac{0.0420}{0.17} \approx 0.25$ Thus, as compute $C$ increases, the optimal model size $N$ scales as $C^{0.55}$, batch size $B$ scales as $C^{0.20}$, and training steps $S$ scale as $C^{0.25}$.
Sources

S1.E7

N∝CαCmin/αN,B∝CαCmin/αB,S∝CαCmin/αS,D=B⋅Sformulae-sequenceproportional-to𝑁superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝑁formulae-sequenceproportional-to𝐵superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝐵formulae-sequenceproportional-to𝑆superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝑆𝐷⋅𝐵𝑆N\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad (1.7)
N\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad
CC
Total compute budget · scalar
NN
Optimal number of model parameters · scalar
BB
Optimal batch size in tokens · scalar
SS
Optimal number of training steps · scalar
DD
Total dataset size in tokens · scalar
αCmin\alpha_{C}^{\rm min}
Minimum compute scaling exponent · scalar
αN\alpha_{N}
Power-law scaling exponent for model size · scalar
αB\alpha_{B}
Power-law scaling exponent for batch size · scalar
αS\alpha_{S}
Power-law scaling exponent for training steps · scalar