Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Basic Power Laws of Scaling

Power-law scaling of loss with dataset size

Source equation

$L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\nobreak\ \nobreak\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\nobreak\ \text{(tokens)}$

This equation models the power-law scaling relationship between the cross-entropy loss $L(D)$ and the dataset size $D$ (measured in tokens) when training a language model that is not bottlenecked by model capacity or compute budget.

Sources

S1.E2

L(D)=(Dc/D)αD;αD∼0.095,Dc∼5.4×1013(tokens)formulae-sequence𝐿𝐷superscriptsubscript𝐷c𝐷subscript𝛼𝐷formulae-sequencesimilar-tosubscript𝛼𝐷0.095similar-tosubscript𝐷c5.4superscript1013(tokens)L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\leavevmode\nobreak\ \text{(tokens)} (1.2)

L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\leavevmode\nobreak\ \text{(tokens)}

$L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\nobreak\ \nobreak\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\nobreak\ \text{(tokens)}$

Sources

S1.E2

L(D)=(Dc/D)αD;αD∼0.095,Dc∼5.4×1013(tokens)formulae-sequence𝐿𝐷superscriptsubscript𝐷c𝐷subscript𝛼𝐷formulae-sequencesimilar-tosubscript𝛼𝐷0.095similar-tosubscript𝐷c5.4superscript1013(tokens)L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\leavevmode\nobreak\ \text{(tokens)} (1.2)

L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\leavevmode\nobreak\ \text{(tokens)}

The power-law relationship indicates that the cross-entropy loss decreases predictably as the dataset size $D$ increases. The parameter $D_{\mathrm{c}}$ represents the critical dataset size scale, and $\alpha_{D}$ is the scaling exponent governing the rate of improvement.

Sources

S1.E2

L(D)=(Dc/D)αD;αD∼0.095,Dc∼5.4×1013(tokens)formulae-sequence𝐿𝐷superscriptsubscript𝐷c𝐷subscript𝛼𝐷formulae-sequencesimilar-tosubscript𝛼𝐷0.095similar-tosubscript𝐷c5.4superscript1013(tokens)L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\leavevmode\nobreak\ \text{(tokens)} (1.2)

L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\leavevmode\nobreak\ \text{(tokens)}

Implementation detail

Illustrative Calculation

Let us calculate the predicted loss $L(D)$ for a dataset size of $D = 10^{10}$ tokens using the empirical parameters:

$D_{\mathrm{c}} = 5.4 \times 10^{13}$
$\alpha_{D} = 0.095$

Step 1: Compute the ratio of critical dataset size to dataset size $\frac{D_{\mathrm{c}}}{D} = \frac{5.4 \times 10^{13}}{10^{10}} = 5400$

Step 2: Apply the power-law exponent $L(D) = (5400)^{0.095} \approx 2.261$

Sources

S1.E2

L(D)=(Dc/D)αD;αD∼0.095,Dc∼5.4×1013(tokens)formulae-sequence𝐿𝐷superscriptsubscript𝐷c𝐷subscript𝛼𝐷formulae-sequencesimilar-tosubscript𝛼𝐷0.095similar-tosubscript𝐷c5.4superscript1013(tokens)L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\leavevmode\nobreak\ \text{(tokens)} (1.2)

L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\leavevmode\nobreak\ \text{(tokens)}

$L(D)$: Cross-entropy loss as a function of dataset size · scalar
$D$: Dataset size in tokens · scalar
$D_{\mathrm{c}}$: Critical dataset size constant · scalar
$\alpha_{D}$: Scaling exponent for dataset size · scalar