Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Transformer Parameter and Compute Scaling

Unified Loss Equation L(N,D)

Source equation

$L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}$

The unified loss equation $L(N,D)$ models the joint dependence of the cross-entropy loss on both the number of non-embedding parameters $N$ and the dataset size $D$ (measured in tokens). It captures how overfitting occurs when either resource is constrained relative to the other.

Sources

S1.E5

L(N,D)=[(NcN)αNαD+DcD]αD𝐿𝑁𝐷superscriptdelimited-[]superscriptsubscript𝑁𝑐𝑁subscript𝛼𝑁subscript𝛼𝐷subscript𝐷𝑐𝐷subscript𝛼𝐷L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} (1.5)

L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}

$L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}$

Sources

S1.E5

L(N,D)=[(NcN)αNαD+DcD]αD𝐿𝑁𝐷superscriptdelimited-[]superscriptsubscript𝑁𝑐𝑁subscript𝛼𝑁subscript𝛼𝐷subscript𝐷𝑐𝐷subscript𝛼𝐷L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} (1.5)

L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}

Where the symbols are defined as follows:

$L(N,D)$ : The predicted cross-entropy loss.
$N$ : The number of model parameters (excluding embeddings).
$D$ : The dataset size in tokens.
$N_c$ : The critical model size parameter.
$D_c$ : The critical dataset size parameter.
$\alpha_N$ : The power-law scaling exponent for model size.
$\alpha_D$ : The power-law scaling exponent for dataset size.

Sources

S1.E5

L(N,D)=[(NcN)αNαD+DcD]αD𝐿𝑁𝐷superscriptdelimited-[]superscriptsubscript𝑁𝑐𝑁subscript𝛼𝑁subscript𝛼𝐷subscript𝐷𝑐𝐷subscript𝛼𝐷L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} (1.5)

L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}

Implementation detail

Illustrative Calculation

Let us compute the loss $L(N,D)$ using a set of simplified parameters:

$N_c = 10^9$ , $D_c = 10^9$
$\alpha_N = 0.08$ , $\alpha_D = 0.08$ (so that $\frac{\alpha_N}{\alpha_D} = 1.0$ )
$N = 10^{10}$ (a model 10 times larger than $N_c$ )
$D = 10^9$ (a dataset equal to $D_c$ )

Step-by-step evaluation:

Calculate the model term: $\left(\frac{N_c}{N}\right)^{\frac{\alpha_N}{\alpha_D}} = \left(\frac{10^9}{10^{10}}\right)^{1.0} = 0.1$
Calculate the dataset term: $\frac{D_c}{D} = \frac{10^9}{10^9} = 1.0$
Sum the terms inside the brackets: $0.1 + 1.0 = 1.1$
Apply the outer exponent $\alpha_D = 0.08$ : $L(N,D) = (1.1)^{0.08} \approx 1.00765$

Sources

S1.E5

L(N,D)=[(NcN)αNαD+DcD]αD𝐿𝑁𝐷superscriptdelimited-[]superscriptsubscript𝑁𝑐𝑁subscript𝛼𝑁subscript𝛼𝐷subscript𝐷𝑐𝐷subscript𝛼𝐷L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} (1.5)

L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}

$L(N,D)$: Cross-entropy loss as a function of model size and dataset size · scalar
$N$: Number of non-embedding model parameters · scalar
$D$: Dataset size in tokens · scalar
$N_c$: Critical model size constant · scalar
$D_c$: Critical dataset size constant · scalar
$\alpha_N$: Power-law scaling exponent for model size · scalar
$\alpha_D$: Power-law scaling exponent for dataset size · scalar