Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Overfitting and the Infinite Data Limit

Joint Scaling Law L(N,D)

Source equation

$L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}$

The joint scaling law $L(N,D)$ models the simultaneous dependence of the cross-entropy loss on both the model parameter count $N$ and the dataset size $D$ . It captures how performance scales when both resources are finite, incorporating the effects of overfitting.

Sources

S1.E5

L(N,D)=[(NcN)αNαD+DcD]αD𝐿𝑁𝐷superscriptdelimited-[]superscriptsubscript𝑁𝑐𝑁subscript𝛼𝑁subscript𝛼𝐷subscript𝐷𝑐𝐷subscript𝛼𝐷L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} (1.5)

L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}

$L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}$

Sources

S1.E5

L(N,D)=[(NcN)αNαD+DcD]αD𝐿𝑁𝐷superscriptdelimited-[]superscriptsubscript𝑁𝑐𝑁subscript𝛼𝑁subscript𝛼𝐷subscript𝐷𝑐𝐷subscript𝛼𝐷L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} (1.5)

L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}

Illustrative Calculation

Let us compute $L(N,D)$ with the following parameters:

Model size $N = 10^8$
Dataset size $D = 10^7$
Critical model size $N_c = 10^{10}$
Critical dataset size $D_c = 10^8$
Scaling exponents $\alpha_N = 0.08$ , $\alpha_D = 0.10$

Step 1: Compute the exponent ratio $\frac{\alpha_N}{\alpha_D} = \frac{0.08}{0.10} = 0.8$

Step 2: Compute the model term $\frac{N_c}{N} = \frac{10^{10}}{10^8} = 100$ $\left(\frac{N_c}{N}\right)^{\frac{\alpha_N}{\alpha_D}} = 100^{0.8} \approx 39.8107$

Step 3: Compute the dataset term $\frac{D_c}{D} = \frac{10^8}{10^7} = 10$

Step 4: Sum the terms and apply the outer exponent $\text{Sum} = 39.8107 + 10 = 49.8107$ $L(N,D) = (49.8107)^{0.10} \approx 1.478$

Sources

S1.E5

L(N,D)=[(NcN)αNαD+DcD]αD𝐿𝑁𝐷superscriptdelimited-[]superscriptsubscript𝑁𝑐𝑁subscript𝛼𝑁subscript𝛼𝐷subscript𝐷𝑐𝐷subscript𝛼𝐷L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} (1.5)

L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}

$L(N,D)$: Cross-entropy loss as a function of model size and dataset size · scalar
$N$: Number of non-embedding model parameters · scalar
$D$: Dataset size in tokens · scalar
$N_c$: Critical model parameter scale constant · scalar
$D_c$: Critical dataset token scale constant · scalar
$\alpha_N$: Power-law scaling exponent for model size · scalar
$\alpha_D$: Power-law scaling exponent for dataset size · scalar