Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

The Infinite Data Limit and Overfitting

Power-law scaling with non-embedding parameters

Source equation

$L(N)=\left(N_{\mathrm{c}}/N\right)^{\alpha_{N}};\nobreak\ \nobreak\ \alpha_{N}\sim 0.076,\quad N_{\mathrm{c}}\sim 8.8\times 10^{13}\nobreak\ \text{(non-embedding parameters)}$

This equation models the empirical power-law scaling of the cross-entropy loss $L(N)$ of a language model as a function of its number of non-embedding parameters $N$ . By excluding embedding parameters, the scaling law exhibits a cleaner power-law behavior over several orders of magnitude.

Sources

S1.E1

L(N)=(Nc/N)αN;αN∼0.076,Nc∼8.8×1013(non-embedding parameters)formulae-sequence𝐿𝑁superscriptsubscript𝑁c𝑁subscript𝛼𝑁formulae-sequencesimilar-tosubscript𝛼𝑁0.076similar-tosubscript𝑁c8.8superscript1013(non-embedding parameters)L(N)=\left(N_{\mathrm{c}}/N\right)^{\alpha_{N}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{N}\sim 0.076,\quad N_{\mathrm{c}}\sim 8.8\times 10^{13}\leavevmode\nobreak\ \text{(non-embedding parameters)} (1.1)

L(N)=\left(N_{\mathrm{c}}/N\right)^{\alpha_{N}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{N}\sim 0.076,\quad N_{\mathrm{c}}\sim 8.8\times 10^{13}\leavevmode\nobreak\ \text{(non-embedding parameters)}

$L(N)=\left(N_{\mathrm{c}}/N\right)^{\alpha_{N}};\nobreak\ \nobreak\ \alpha_{N}\sim 0.076,\quad N_{\mathrm{c}}\sim 8.8\times 10^{13}\nobreak\ \text{(non-embedding parameters)}$

Sources

S1.E1

L(N)=(Nc/N)αN;αN∼0.076,Nc∼8.8×1013(non-embedding parameters)formulae-sequence𝐿𝑁superscriptsubscript𝑁c𝑁subscript𝛼𝑁formulae-sequencesimilar-tosubscript𝛼𝑁0.076similar-tosubscript𝑁c8.8superscript1013(non-embedding parameters)L(N)=\left(N_{\mathrm{c}}/N\right)^{\alpha_{N}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{N}\sim 0.076,\quad N_{\mathrm{c}}\sim 8.8\times 10^{13}\leavevmode\nobreak\ \text{(non-embedding parameters)} (1.1)

L(N)=\left(N_{\mathrm{c}}/N\right)^{\alpha_{N}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{N}\sim 0.076,\quad N_{\mathrm{c}}\sim 8.8\times 10^{13}\leavevmode\nobreak\ \text{(non-embedding parameters)}

Illustrative Calculation

Let us calculate the predicted loss $L(N)$ for a model with $N = 10^9$ non-embedding parameters (a 1-billion parameter model):

Inputs:
- $N = 10^9$
- $N_{\mathrm{c}} = 8.8 \times 10^{13}$
- $\alpha_N = 0.076$
Ratio Calculation: $\frac{N_{\mathrm{c}}}{N} = \frac{8.8 \times 10^{13}}{10^9} = 8.8 \times 10^4 = 88000$
Power-law Exponentiation: $L(N) = (88000)^{0.076} \approx 2.373$

Sources

S1.E1

L(N)=(Nc/N)αN;αN∼0.076,Nc∼8.8×1013(non-embedding parameters)formulae-sequence𝐿𝑁superscriptsubscript𝑁c𝑁subscript𝛼𝑁formulae-sequencesimilar-tosubscript𝛼𝑁0.076similar-tosubscript𝑁c8.8superscript1013(non-embedding parameters)L(N)=\left(N_{\mathrm{c}}/N\right)^{\alpha_{N}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{N}\sim 0.076,\quad N_{\mathrm{c}}\sim 8.8\times 10^{13}\leavevmode\nobreak\ \text{(non-embedding parameters)} (1.1)

L(N)=\left(N_{\mathrm{c}}/N\right)^{\alpha_{N}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{N}\sim 0.076,\quad N_{\mathrm{c}}\sim 8.8\times 10^{13}\leavevmode\nobreak\ \text{(non-embedding parameters)}

$L(N)$: Cross-entropy loss · scalar
$N$: Number of non-embedding parameters · scalar
$N_{\mathrm{c}}$: Scale parameter constant · scalar
$\alpha_{N}$: Power-law scaling exponent · scalar