Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Basic Power Laws of Scaling

Power-law scaling of loss with model size

Source equation

$L(N)=\left(N_{\mathrm{c}}/N\right)^{\alpha_{N}};\nobreak\ \nobreak\ \alpha_{N}\sim 0.076,\quad N_{\mathrm{c}}\sim 8.8\times 10^{13}\nobreak\ \text{(non-embedding parameters)}$

This equation models the power-law scaling relationship between the cross-entropy loss $L(N)$ and the number of non-embedding parameters $N$ of a neural language model, assuming performance is not bottlenecked by dataset size or compute limits.

Sources

S1.E1

L(N)=(Nc/N)αN;αN∼0.076,Nc∼8.8×1013(non-embedding parameters)formulae-sequence𝐿𝑁superscriptsubscript𝑁c𝑁subscript𝛼𝑁formulae-sequencesimilar-tosubscript𝛼𝑁0.076similar-tosubscript𝑁c8.8superscript1013(non-embedding parameters)L(N)=\left(N_{\mathrm{c}}/N\right)^{\alpha_{N}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{N}\sim 0.076,\quad N_{\mathrm{c}}\sim 8.8\times 10^{13}\leavevmode\nobreak\ \text{(non-embedding parameters)} (1.1)

L(N)=\left(N_{\mathrm{c}}/N\right)^{\alpha_{N}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{N}\sim 0.076,\quad N_{\mathrm{c}}\sim 8.8\times 10^{13}\leavevmode\nobreak\ \text{(non-embedding parameters)}

$L(N)=\left(N_{\mathrm{c}}/N\right)^{\alpha_{N}};\nobreak\ \nobreak\ \alpha_{N}\sim 0.076,\quad N_{\mathrm{c}}\sim 8.8\times 10^{13}\nobreak\ \text{(non-embedding parameters)}$

Sources

S1.E1

L(N)=(Nc/N)αN;αN∼0.076,Nc∼8.8×1013(non-embedding parameters)formulae-sequence𝐿𝑁superscriptsubscript𝑁c𝑁subscript𝛼𝑁formulae-sequencesimilar-tosubscript𝛼𝑁0.076similar-tosubscript𝑁c8.8superscript1013(non-embedding parameters)L(N)=\left(N_{\mathrm{c}}/N\right)^{\alpha_{N}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{N}\sim 0.076,\quad N_{\mathrm{c}}\sim 8.8\times 10^{13}\leavevmode\nobreak\ \text{(non-embedding parameters)} (1.1)

L(N)=\left(N_{\mathrm{c}}/N\right)^{\alpha_{N}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{N}\sim 0.076,\quad N_{\mathrm{c}}\sim 8.8\times 10^{13}\leavevmode\nobreak\ \text{(non-embedding parameters)}

Deep dive

Illustrative Calculation

Let us calculate the predicted cross-entropy loss $L(N)$ for a model with $N = 10^9$ non-embedding parameters (a 1-billion parameter model).

Using the empirical constants:

$N_{\mathrm{c}} = 8.8 \times 10^{13}$
$\alpha_{N} = 0.076$

We compute the ratio: $\frac{N_{\mathrm{c}}}{N} = \frac{8.8 \times 10^{13}}{10^9} = 8.8 \times 10^4 = 88000$

Now, we raise this ratio to the power of $\alpha_{N}$ : $L(10^9) = (88000)^{0.076} \approx 2.372$

Sources

S1.E1

L(N)=(Nc/N)αN;αN∼0.076,Nc∼8.8×1013(non-embedding parameters)formulae-sequence𝐿𝑁superscriptsubscript𝑁c𝑁subscript𝛼𝑁formulae-sequencesimilar-tosubscript𝛼𝑁0.076similar-tosubscript𝑁c8.8superscript1013(non-embedding parameters)L(N)=\left(N_{\mathrm{c}}/N\right)^{\alpha_{N}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{N}\sim 0.076,\quad N_{\mathrm{c}}\sim 8.8\times 10^{13}\leavevmode\nobreak\ \text{(non-embedding parameters)} (1.1)

L(N)=\left(N_{\mathrm{c}}/N\right)^{\alpha_{N}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{N}\sim 0.076,\quad N_{\mathrm{c}}\sim 8.8\times 10^{13}\leavevmode\nobreak\ \text{(non-embedding parameters)}

$L(N)$: Cross-entropy loss · scalar
$N$: Number of non-embedding parameters in the model · scalar
$N_c$: Scaling parameter constant representing the scale parameter for model size · scalar
$\alpha_N$: Power-law scaling exponent for model size · scalar