Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Transformer Parameter and Compute Scaling

Optimal Compute Allocation Scaling Relations

Source equation

$N\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad$

This set of scaling relations determines how to optimally allocate an increased compute budget $C$ among the model size (number of non-embedding parameters $N$ ), the batch size $B$ , and the number of training steps $S$ . The total number of tokens processed during training is given by $D = B \cdot S$ .

Sources

S1.E7

N∝CαCmin/αN,B∝CαCmin/αB,S∝CαCmin/αS,D=B⋅Sformulae-sequenceproportional-to𝑁superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝑁formulae-sequenceproportional-to𝐵superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝐵formulae-sequenceproportional-to𝑆superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝑆𝐷⋅𝐵𝑆N\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad (1.7)

N\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad

$N\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad$

Sources

S1.E7

N∝CαCmin/αN,B∝CαCmin/αB,S∝CαCmin/αS,D=B⋅Sformulae-sequenceproportional-to𝑁superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝑁formulae-sequenceproportional-to𝐵superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝐵formulae-sequenceproportional-to𝑆superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝑆𝐷⋅𝐵𝑆N\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad (1.7)

N\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad

Deep dive

Illustrative Calculation

Let us compute the scaling exponents given the empirical values from the paper:

$\alpha_N = 0.076$
$\alpha_B = 0.21$
$\alpha_S = 0.17$

First, we find the minimum scaling exponent $\alpha_C^{\rm min}$ using the relation: $\alpha_C^{\rm min} = \left( \frac{1}{\alpha_N} + \frac{1}{\alpha_B} + \frac{1}{\alpha_S} \right)^{-1}$

Let's calculate the terms:

$1/\alpha_N = 1 / 0.076 \approx 13.158$
$1/\alpha_B = 1 / 0.21 \approx 4.762$
$1/\alpha_S = 1 / 0.17 \approx 5.882$

Summing these values: $\sum \approx 13.158 + 4.762 + 5.882 = 23.802$

Thus, the minimum exponent is: $\alpha_C^{\rm min} = 1 / 23.802 \approx 0.0420$

Now, we compute the scaling exponents for each resource allocation:

For model size $N$ : $\alpha_C^{\rm min} / \alpha_N \approx 0.0420 / 0.076 \approx 0.55$
For batch size $B$ : $\alpha_C^{\rm min} / \alpha_B \approx 0.0420 / 0.21 \approx 0.20$
For steps $S$ : $\alpha_C^{\rm min} / \alpha_S \approx 0.0420 / 0.17 \approx 0.25$

This shows that as the compute budget $C$ increases, the optimal allocation scales the model size $N$ fastest ( $N \propto C^{0.55}$ ), followed by training steps ( $S \propto C^{0.25}$ ) and batch size ( $B \propto C^{0.20}$ ).

Sources

S1.E7

N∝CαCmin/αN,B∝CαCmin/αB,S∝CαCmin/αS,D=B⋅Sformulae-sequenceproportional-to𝑁superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝑁formulae-sequenceproportional-to𝐵superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝐵formulae-sequenceproportional-to𝑆superscript𝐶superscriptsubscript𝛼𝐶minsubscript𝛼𝑆𝐷⋅𝐵𝑆N\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad (1.7)

N\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad

$C$: Total compute budget · scalar
$N$: Optimal model size (number of non-embedding parameters) · scalar
$B$: Optimal batch size · scalar
$S$: Optimal number of training steps · scalar
$D$: Total dataset size in tokens · scalar
$alpha_C_min$: Minimum scaling exponent for compute · scalar
$alpha_N$: Power-law scaling exponent for model size · scalar
$alpha_B$: Power-law scaling exponent for batch size · scalar
$alpha_S$: Power-law scaling exponent for training steps · scalar