Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Overfitting and the Infinite Data Limit

Joint Scaling Law L(N,D)

Joint Scaling Law L(N,D)

Source equation

L(N,D)=[(NcN)αNαD+DcD]αDL(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}

The joint scaling law L(N,D)L(N,D) models the simultaneous dependence of the cross-entropy loss on both the model parameter count NN and the dataset size DD. It captures how performance scales when both resources are finite, incorporating the effects of overfitting.

Sources

S1.E5

L​(N,D)=[(NcN)αNαD+DcD]αD𝐿𝑁𝐷superscriptdelimited-[]superscriptsubscript𝑁𝑐𝑁subscript𝛼𝑁subscript𝛼𝐷subscript𝐷𝑐𝐷subscript𝛼𝐷L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} (1.5)
L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}

L(N,D)=[(NcN)αNαD+DcD]αDL(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}

Sources

S1.E5

L​(N,D)=[(NcN)αNαD+DcD]αD𝐿𝑁𝐷superscriptdelimited-[]superscriptsubscript𝑁𝑐𝑁subscript𝛼𝑁subscript𝛼𝐷subscript𝐷𝑐𝐷subscript𝛼𝐷L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} (1.5)
L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}

Illustrative Calculation

Let us compute L(N,D)L(N,D) with the following parameters:

  • Model size N=108N = 10^8
  • Dataset size D=107D = 10^7
  • Critical model size Nc=1010N_c = 10^{10}
  • Critical dataset size Dc=108D_c = 10^8
  • Scaling exponents αN=0.08\alpha_N = 0.08, αD=0.10\alpha_D = 0.10

Step 1: Compute the exponent ratio αNαD=0.080.10=0.8\frac{\alpha_N}{\alpha_D} = \frac{0.08}{0.10} = 0.8

Step 2: Compute the model term NcN=1010108=100\frac{N_c}{N} = \frac{10^{10}}{10^8} = 100 (NcN)αNαD=1000.8≈39.8107\left(\frac{N_c}{N}\right)^{\frac{\alpha_N}{\alpha_D}} = 100^{0.8} \approx 39.8107

Step 3: Compute the dataset term DcD=108107=10\frac{D_c}{D} = \frac{10^8}{10^7} = 10

Step 4: Sum the terms and apply the outer exponent Sum=39.8107+10=49.8107\text{Sum} = 39.8107 + 10 = 49.8107 L(N,D)=(49.8107)0.10≈1.478L(N,D) = (49.8107)^{0.10} \approx 1.478

Sources

S1.E5

L​(N,D)=[(NcN)αNαD+DcD]αD𝐿𝑁𝐷superscriptdelimited-[]superscriptsubscript𝑁𝑐𝑁subscript𝛼𝑁subscript𝛼𝐷subscript𝐷𝑐𝐷subscript𝛼𝐷L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} (1.5)
L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}
L(N,D)L(N,D)
Cross-entropy loss as a function of model size and dataset size · scalar
NN
Number of non-embedding model parameters · scalar
DD
Dataset size in tokens · scalar
NcN_c
Critical model parameter scale constant · scalar
DcD_c
Critical dataset token scale constant · scalar
αN\alpha_N
Power-law scaling exponent for model size · scalar
αD\alpha_D
Power-law scaling exponent for dataset size · scalar