Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

The Infinite Data Limit and Overfitting

Power-law scaling with minimum compute

Source equation

$L(C_{\rm min})=\left(C_{\mathrm{c}}^{\rm min}/C_{\rm min}\right)^{\alpha_{C}^{\rm min}};\nobreak\ \nobreak\ \alpha_{C}^{\rm min}\sim 0.050,\quad C_{\mathrm{c}}^{\rm min}\sim 3.1\times 10^{8}\nobreak\ \text{(PF-days)}$

This equation describes the power-law scaling of the cross-entropy loss $L$ as a function of the minimum training compute $C_{\rm min}$ (measured in PF-days) required to reach a given performance level.

Sources

S1.E3

L(Cmin)=(Ccmin/Cmin)αCmin;αCmin∼0.050,Ccmin∼3.1×108(PF-days)formulae-sequence𝐿subscript𝐶minsuperscriptsuperscriptsubscript𝐶cminsubscript𝐶minsuperscriptsubscript𝛼𝐶minformulae-sequencesimilar-tosuperscriptsubscript𝛼𝐶min0.050similar-tosuperscriptsubscript𝐶cmin3.1superscript108(PF-days)L(C_{\rm min})=\left(C_{\mathrm{c}}^{\rm min}/C_{\rm min}\right)^{\alpha_{C}^{\rm min}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{C}^{\rm min}\sim 0.050,\quad C_{\mathrm{c}}^{\rm min}\sim 3.1\times 10^{8}\leavevmode\nobreak\ \text{(PF-days)} (1.3)

L(C_{\rm min})=\left(C_{\mathrm{c}}^{\rm min}/C_{\rm min}\right)^{\alpha_{C}^{\rm min}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{C}^{\rm min}\sim 0.050,\quad C_{\mathrm{c}}^{\rm min}\sim 3.1\times 10^{8}\leavevmode\nobreak\ \text{(PF-days)}

Sources

S1.E3

L(Cmin)=(Ccmin/Cmin)αCmin;αCmin∼0.050,Ccmin∼3.1×108(PF-days)formulae-sequence𝐿subscript𝐶minsuperscriptsuperscriptsubscript𝐶cminsubscript𝐶minsuperscriptsubscript𝛼𝐶minformulae-sequencesimilar-tosuperscriptsubscript𝛼𝐶min0.050similar-tosuperscriptsubscript𝐶cmin3.1superscript108(PF-days)L(C_{\rm min})=\left(C_{\mathrm{c}}^{\rm min}/C_{\rm min}\right)^{\alpha_{C}^{\rm min}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{C}^{\rm min}\sim 0.050,\quad C_{\mathrm{c}}^{\rm min}\sim 3.1\times 10^{8}\leavevmode\nobreak\ \text{(PF-days)} (1.3)

L(C_{\rm min})=\left(C_{\mathrm{c}}^{\rm min}/C_{\rm min}\right)^{\alpha_{C}^{\rm min}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{C}^{\rm min}\sim 0.050,\quad C_{\mathrm{c}}^{\rm min}\sim 3.1\times 10^{8}\leavevmode\nobreak\ \text{(PF-days)}

Illustrative Calculation

Let us compute the loss $L(C_{\rm min})$ for a given minimum compute budget of $C_{\rm min} = 10^4$ PF-days.

Using the empirical parameters:

$C_{\mathrm{c}}^{\rm min} = 3.1 \times 10^8$ PF-days
$\alpha_{C}^{\rm min} = 0.050$

We calculate:

The ratio: $\frac{C_{\mathrm{c}}^{\rm min}}{C_{\rm min}} = \frac{3.1 \times 10^8}{10^4} = 3.1 \times 10^4 = 31000$
The power-law exponentiation: $31000^{0.050} \approx 1.675$

Thus, the predicted loss $L(10^4)$ is approximately $1.675$ nats.

Sources

S1.E3

L(Cmin)=(Ccmin/Cmin)αCmin;αCmin∼0.050,Ccmin∼3.1×108(PF-days)formulae-sequence𝐿subscript𝐶minsuperscriptsuperscriptsubscript𝐶cminsubscript𝐶minsuperscriptsubscript𝛼𝐶minformulae-sequencesimilar-tosuperscriptsubscript𝛼𝐶min0.050similar-tosuperscriptsubscript𝐶cmin3.1superscript108(PF-days)L(C_{\rm min})=\left(C_{\mathrm{c}}^{\rm min}/C_{\rm min}\right)^{\alpha_{C}^{\rm min}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{C}^{\rm min}\sim 0.050,\quad C_{\mathrm{c}}^{\rm min}\sim 3.1\times 10^{8}\leavevmode\nobreak\ \text{(PF-days)} (1.3)

L(C_{\rm min})=\left(C_{\mathrm{c}}^{\rm min}/C_{\rm min}\right)^{\alpha_{C}^{\rm min}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{C}^{\rm min}\sim 0.050,\quad C_{\mathrm{c}}^{\rm min}\sim 3.1\times 10^{8}\leavevmode\nobreak\ \text{(PF-days)}

$L(C_{\rm min})$: Cross-entropy loss as a function of minimum compute · scalar
$C_{\rm min}$: Minimum compute budget used for training · scalar
$C_{\mathrm{c}}^{\rm min}$: Scale parameter for the minimum compute · scalar
$\alpha_{C}^{\rm min}$: Power-law scaling exponent for the minimum compute · scalar