Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

The Infinite Data Limit and Overfitting

Unified scaling law for parameters and dataset size

Source equation

$L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}$

The unified scaling law models the cross-entropy loss $L(N,D)$ as a joint function of the number of non-embedding parameters $N$ and the dataset size $D$ (measured in tokens). This formulation captures how performance scales when both resources are finite, predicting the onset of overfitting when one resource is bottlenecked relative to the other.

Sources

S1.E5

L(N,D)=[(NcN)αNαD+DcD]αD𝐿𝑁𝐷superscriptdelimited-[]superscriptsubscript𝑁𝑐𝑁subscript𝛼𝑁subscript𝛼𝐷subscript𝐷𝑐𝐷subscript𝛼𝐷L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} (1.5)

L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}

$L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}$

Sources

S1.E5

L(N,D)=[(NcN)αNαD+DcD]αD𝐿𝑁𝐷superscriptdelimited-[]superscriptsubscript𝑁𝑐𝑁subscript𝛼𝑁subscript𝛼𝐷subscript𝐷𝑐𝐷subscript𝛼𝐷L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} (1.5)

L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}

Symbol Definitions

$L(N,D)$ : The cross-entropy loss, representing the model's performance.
$N$ : The number of model parameters (excluding embeddings).
$D$ : The dataset size in tokens.
$N_c$ : The critical parameter scale constant.
$D_c$ : The critical dataset scale constant.
$\alpha_N$ : The power-law scaling exponent for model size $N$ .
$\alpha_D$ : The power-law scaling exponent for dataset size $D$ .

Sources

S1.E5

L(N,D)=[(NcN)αNαD+DcD]αD𝐿𝑁𝐷superscriptdelimited-[]superscriptsubscript𝑁𝑐𝑁subscript𝛼𝑁subscript𝛼𝐷subscript𝐷𝑐𝐷subscript𝛼𝐷L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} (1.5)

L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}

Implementation detail

Illustrative Calculation

Let us compute the loss $L(N,D)$ for a specific set of parameters:

$N_c = 10^5$ , $D_c = 10^6$
$\alpha_N = 0.08$ , $\alpha_D = 0.16$
Ratio of exponents: $\frac{\alpha_N}{\alpha_D} = \frac{0.08}{0.16} = 0.5$

We evaluate at $N = 10^7$ and $D = 10^8$ :

Parameter term ratio: $\frac{N_c}{N} = \frac{10^5}{10^7} = 10^{-2} = 0.01$
Exponentiated parameter term: $\left(\frac{N_c}{N}\right)^{\frac{\alpha_N}{\alpha_D}} = (0.01)^{0.5} = 0.1$
Dataset term: $\frac{D_c}{D} = \frac{10^6}{10^8} = 10^{-2} = 0.01$
Sum of terms: $0.1 + 0.01 = 0.11$
Final Loss: $L(N,D) = (0.11)^{0.16} \approx 0.7015$

Sources

S1.E5

L(N,D)=[(NcN)αNαD+DcD]αD𝐿𝑁𝐷superscriptdelimited-[]superscriptsubscript𝑁𝑐𝑁subscript𝛼𝑁subscript𝛼𝐷subscript𝐷𝑐𝐷subscript𝛼𝐷L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} (1.5)

L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}

$L(N,D)$: Cross-entropy loss · scalar
$N$: Number of non-embedding model parameters · scalar
$D$: Dataset size in tokens · scalar
$N_c$: Critical parameter scale constant · scalar
$D_c$: Critical dataset scale constant · scalar
$alpha_N$: Power-law scaling exponent for model size · scalar
$alpha_D$: Power-law scaling exponent for dataset size · scalar