Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

The Infinite Data Limit and Overfitting

Optimal compute allocation scaling

Source equation

$N\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad$

This equation describes the optimal scaling of model parameters $N$ , batch size $B$ , and training steps $S$ as power-law functions of the total compute budget $C$ . It also defines the total dataset size $D$ in tokens as the product of batch size and training steps.

Sources

S1.E7

N∝CαCmin/αN,B∝CαCmin/αB,S∝CαCmin/αS,D=B⋅Sformulae-sequenceproportional-to𝑁superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝑁formulae-sequenceproportional-to𝐵superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝐵formulae-sequenceproportional-to𝑆superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝑆𝐷⋅𝐵𝑆N\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad (1.7)

N\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad

$N\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad$

Sources

S1.E7

N∝CαCmin/αN,B∝CαCmin/αB,S∝CαCmin/αS,D=B⋅Sformulae-sequenceproportional-to𝑁superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝑁formulae-sequenceproportional-to𝐵superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝐵formulae-sequenceproportional-to𝑆superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝑆𝐷⋅𝐵𝑆N\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad (1.7)

N\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad

Implementation detail

Illustrative Calculation

Let us calculate the scaling exponents given the following empirical values:

$\alpha_N = 0.076$
$\alpha_B = 0.21$
$\alpha_S = 0.17$

First, we compute the minimum compute scaling exponent $\alpha_C^{\rm min}$ using the relation: $\frac{1}{\alpha_C^{\rm min}} = \frac{1}{\alpha_N} + \frac{1}{\alpha_B} + \frac{1}{\alpha_S}$

$\frac{1}{\alpha_C^{\rm min}} = \frac{1}{0.076} + \frac{1}{0.21} + \frac{1}{0.17} \approx 13.158 + 4.762 + 5.882 = 23.802$

$\alpha_C^{\rm min} \approx \frac{1}{23.802} \approx 0.0420$

Now, we compute the power-law exponents for $N$ , $B$ , and $S$ :

Exponent for $N$ : $a_N = \frac{\alpha_C^{\rm min}}{\alpha_N} \approx \frac{0.0420}{0.076} \approx 0.55$
Exponent for $B$ : $a_B = \frac{\alpha_C^{\rm min}}{\alpha_B} \approx \frac{0.0420}{0.21} \approx 0.20$
Exponent for $S$ : $a_S = \frac{\alpha_C^{\rm min}}{\alpha_S} \approx \frac{0.0420}{0.17} \approx 0.25$

Thus, as compute $C$ increases, the optimal parameters scale as $N \propto C^{0.55}$ , batch size as $B \propto C^{0.20}$ , and steps as $S \propto C^{0.25}$ . The total dataset size $D = B \cdot S$ scales as $D \propto C^{0.20 + 0.25} = C^{0.45}$ .

Sources

S1.E7

N∝CαCmin/αN,B∝CαCmin/αB,S∝CαCmin/αS,D=B⋅Sformulae-sequenceproportional-to𝑁superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝑁formulae-sequenceproportional-to𝐵superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝐵formulae-sequenceproportional-to𝑆superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝑆𝐷⋅𝐵𝑆N\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad (1.7)

N\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad

$C$: Total compute budget · scalar
$N$: Number of model parameters (excluding embeddings) · scalar
$B$: Critical batch size · scalar
$S$: Number of training steps · scalar
$D$: Total dataset size in tokens · scalar
$\alpha_{C}^{\rm min}$: Minimum compute scaling exponent · scalar
$\alpha_{N}$: Power-law scaling exponent for model parameters · scalar
$\alpha_{B}$: Power-law scaling exponent for batch size · scalar
$\alpha_{S}$: Power-law scaling exponent for training steps · scalar