Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Overfitting and Joint Scaling L(N, D)

Joint scaling law of loss with model and dataset size

Source equation

$L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}$

The joint scaling law $L(N, D)$ models the cross-entropy loss of a language model as a function of both the number of non-embedding parameters $N$ and the dataset size $D$ (measured in tokens). This formulation captures how performance scales when both resources are finite, describing the transition into the overfitting regime.

Sources

S1.E5

L(N,D)=[(NcN)αNαD+DcD]αD𝐿𝑁𝐷superscriptdelimited-[]superscriptsubscript𝑁𝑐𝑁subscript𝛼𝑁subscript𝛼𝐷subscript𝐷𝑐𝐷subscript𝛼𝐷L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} (1.5)

L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}

$L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}$

Sources

S1.E5

L(N,D)=[(NcN)αNαD+DcD]αD𝐿𝑁𝐷superscriptdelimited-[]superscriptsubscript𝑁𝑐𝑁subscript𝛼𝑁subscript𝛼𝐷subscript𝐷𝑐𝐷subscript𝛼𝐷L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} (1.5)

L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}

This equation interpolates between two regimes:

Data-unconstrained regime ( $D \to \infty$ ): The loss simplifies to the power law $L(N) \approx \left(N_c / N\right)^{\alpha_N}$ .
Model-unconstrained regime ( $N \to \infty$ ): The loss simplifies to the power law $L(D) \approx \left(D_c / D\right)^{\alpha_D}$ .

Sources

S1.E5

L(N,D)=[(NcN)αNαD+DcD]αD𝐿𝑁𝐷superscriptdelimited-[]superscriptsubscript𝑁𝑐𝑁subscript𝛼𝑁subscript𝛼𝐷subscript𝐷𝑐𝐷subscript𝛼𝐷L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} (1.5)

L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}

Implementation detail

Illustrative Calculation

Let us compute $L(N, D)$ given the following parameters:

$N_c = 10^8$ , $D_c = 10^9$
$\alpha_N = 0.08$ , $\alpha_D = 0.10$
Model size $N = 10^8$
Dataset size $D = 10^9$

Step 1: Compute the exponent ratio $\frac{\alpha_N}{\alpha_D} = \frac{0.08}{0.10} = 0.8$

Step 2: Compute the model term $\left(\frac{N_c}{N}\right)^{0.8} = \left(\frac{10^8}{10^8}\right)^{0.8} = 1.0^{0.8} = 1.0$

Step 3: Compute the dataset term $\frac{D_c}{D} = \frac{10^9}{10^9} = 1.0$

Step 4: Sum the terms and apply the outer exponent $L(N, D) = [1.0 + 1.0]^{0.10} = 2^{0.10} \approx 1.0718$

Sources

S1.E5

L(N,D)=[(NcN)αNαD+DcD]αD𝐿𝑁𝐷superscriptdelimited-[]superscriptsubscript𝑁𝑐𝑁subscript𝛼𝑁subscript𝛼𝐷subscript𝐷𝑐𝐷subscript𝛼𝐷L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} (1.5)

L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}

$L(N, D)$: Cross-entropy loss · scalar
$N$: Number of non-embedding model parameters · scalar
$D$: Dataset size in tokens · scalar
$N_c$: Critical scale parameter for model size · scalar
$D_c$: Critical scale parameter for dataset size · scalar
$\alpha_N$: Power-law scaling exponent for model size · scalar
$\alpha_D$: Power-law scaling exponent for dataset size · scalar