Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

The Infinite Data Limit and Overfitting

Power-law scaling with dataset size

Source equation

$L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\nobreak\ \nobreak\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\nobreak\ \text{(tokens)}$

This equation models the power-law scaling of language model performance (measured by cross-entropy loss) as a function of the dataset size $D$ (in tokens), assuming that the model capacity and training compute are not bottlenecking performance.

Sources

S1.E2

L(D)=(Dc/D)αD;αD∼0.095,Dc∼5.4×1013(tokens)formulae-sequence𝐿𝐷superscriptsubscript𝐷c𝐷subscript𝛼𝐷formulae-sequencesimilar-tosubscript𝛼𝐷0.095similar-tosubscript𝐷c5.4superscript1013(tokens)L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\leavevmode\nobreak\ \text{(tokens)} (1.2)

L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\leavevmode\nobreak\ \text{(tokens)}

$L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\nobreak\ \nobreak\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\nobreak\ \text{(tokens)}$

Sources

S1.E2

L(D)=(Dc/D)αD;αD∼0.095,Dc∼5.4×1013(tokens)formulae-sequence𝐿𝐷superscriptsubscript𝐷c𝐷subscript𝛼𝐷formulae-sequencesimilar-tosubscript𝛼𝐷0.095similar-tosubscript𝐷c5.4superscript1013(tokens)L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\leavevmode\nobreak\ \text{(tokens)} (1.2)

L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\leavevmode\nobreak\ \text{(tokens)}

Illustrative Calculation

Let us calculate the predicted cross-entropy loss $L(D)$ for a dataset size of $D = 10^9$ tokens.

Using the empirical constants:

$D_{\mathrm{c}} = 5.4 \times 10^{13}$
$\alpha_{D} = 0.095$

We compute:

The ratio: $\frac{D_{\mathrm{c}}}{D} = \frac{5.4 \times 10^{13}}{10^9} = 54,000$
The power-law scaling: $L(D) = (54,000)^{0.095} \approx 2.808$

Sources

S1.E2

L(D)=(Dc/D)αD;αD∼0.095,Dc∼5.4×1013(tokens)formulae-sequence𝐿𝐷superscriptsubscript𝐷c𝐷subscript𝛼𝐷formulae-sequencesimilar-tosubscript𝛼𝐷0.095similar-tosubscript𝐷c5.4superscript1013(tokens)L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\leavevmode\nobreak\ \text{(tokens)} (1.2)

L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\leavevmode\nobreak\ \text{(tokens)}

$L(D)$: Cross-entropy loss as a function of dataset size · scalar
$D$: Dataset size in tokens · scalar
$D_{\mathrm{c}}$: Critical dataset size constant · scalar
$\alpha_{D}$: Power-law scaling exponent for dataset size · scalar