Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Basic Power Laws of Scaling

Power-law scaling of loss with minimum compute

Source equation

$L(C_{\rm min})=\left(C_{\mathrm{c}}^{\rm min}/C_{\rm min}\right)^{\alpha_{C}^{\rm min}};\nobreak\ \nobreak\ \alpha_{C}^{\rm min}\sim 0.050,\quad C_{\mathrm{c}}^{\rm min}\sim 3.1\times 10^{8}\nobreak\ \text{(PF-days)}$

This equation models the power-law scaling of the cross-entropy loss $L$ as a function of the minimum compute $C_{\rm min}$ required to reach that loss, assuming training is not bottlenecked by dataset size or model capacity.

Sources

S1.E3

L(Cmin)=(Ccmin/Cmin)αCmin;αCmin∼0.050,Ccmin∼3.1×108(PF-days)formulae-sequence𝐿subscript𝐶minsuperscriptsuperscriptsubscript𝐶cminsubscript𝐶minsuperscriptsubscript𝛼𝐶minformulae-sequencesimilar-tosuperscriptsubscript𝛼𝐶min0.050similar-tosuperscriptsubscript𝐶cmin3.1superscript108(PF-days)L(C_{\rm min})=\left(C_{\mathrm{c}}^{\rm min}/C_{\rm min}\right)^{\alpha_{C}^{\rm min}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{C}^{\rm min}\sim 0.050,\quad C_{\mathrm{c}}^{\rm min}\sim 3.1\times 10^{8}\leavevmode\nobreak\ \text{(PF-days)} (1.3)

L(C_{\rm min})=\left(C_{\mathrm{c}}^{\rm min}/C_{\rm min}\right)^{\alpha_{C}^{\rm min}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{C}^{\rm min}\sim 0.050,\quad C_{\mathrm{c}}^{\rm min}\sim 3.1\times 10^{8}\leavevmode\nobreak\ \text{(PF-days)}

Sources

S1.E3

L(Cmin)=(Ccmin/Cmin)αCmin;αCmin∼0.050,Ccmin∼3.1×108(PF-days)formulae-sequence𝐿subscript𝐶minsuperscriptsuperscriptsubscript𝐶cminsubscript𝐶minsuperscriptsubscript𝛼𝐶minformulae-sequencesimilar-tosuperscriptsubscript𝛼𝐶min0.050similar-tosuperscriptsubscript𝐶cmin3.1superscript108(PF-days)L(C_{\rm min})=\left(C_{\mathrm{c}}^{\rm min}/C_{\rm min}\right)^{\alpha_{C}^{\rm min}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{C}^{\rm min}\sim 0.050,\quad C_{\mathrm{c}}^{\rm min}\sim 3.1\times 10^{8}\leavevmode\nobreak\ \text{(PF-days)} (1.3)

L(C_{\rm min})=\left(C_{\mathrm{c}}^{\rm min}/C_{\rm min}\right)^{\alpha_{C}^{\rm min}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{C}^{\rm min}\sim 0.050,\quad C_{\mathrm{c}}^{\rm min}\sim 3.1\times 10^{8}\leavevmode\nobreak\ \text{(PF-days)}

Illustrative Calculation

Let's calculate the predicted loss $L(C_{\rm min})$ for a given minimum compute value.

Inputs:

$C_{\rm min} = 1.0 \times 10^{12}$ PF-days
$C_{\mathrm{c}}^{\rm min} = 3.1 \times 10^{8}$ PF-days
$\alpha_{C}^{\rm min} = 0.050$

Step-by-step derivation:

Compute the ratio of the critical compute to the given compute: $\frac{C_{\mathrm{c}}^{\rm min}}{C_{\rm min}} = \frac{3.1 \times 10^{8}}{1.0 \times 10^{12}} = 3.1 \times 10^{-4} = 0.00031$
Raise this ratio to the power of $\alpha_{C}^{\rm min}$ : $L(C_{\rm min}) = (0.00031)^{0.050} \approx 0.6676$

Sources

S1.E3

L(Cmin)=(Ccmin/Cmin)αCmin;αCmin∼0.050,Ccmin∼3.1×108(PF-days)formulae-sequence𝐿subscript𝐶minsuperscriptsuperscriptsubscript𝐶cminsubscript𝐶minsuperscriptsubscript𝛼𝐶minformulae-sequencesimilar-tosuperscriptsubscript𝛼𝐶min0.050similar-tosuperscriptsubscript𝐶cmin3.1superscript108(PF-days)L(C_{\rm min})=\left(C_{\mathrm{c}}^{\rm min}/C_{\rm min}\right)^{\alpha_{C}^{\rm min}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{C}^{\rm min}\sim 0.050,\quad C_{\mathrm{c}}^{\rm min}\sim 3.1\times 10^{8}\leavevmode\nobreak\ \text{(PF-days)} (1.3)

L(C_{\rm min})=\left(C_{\mathrm{c}}^{\rm min}/C_{\rm min}\right)^{\alpha_{C}^{\rm min}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{C}^{\rm min}\sim 0.050,\quad C_{\mathrm{c}}^{\rm min}\sim 3.1\times 10^{8}\leavevmode\nobreak\ \text{(PF-days)}

$L(C_{\rm min})$: Cross-entropy loss as a function of minimum compute · scalar
$C_{\rm min}$: Minimum compute required to achieve a given loss, measured in PF-days · scalar
$C_{\mathrm{c}}^{\rm min}$: Critical compute scale constant, approximately 3.1e8 PF-days · scalar
$\alpha_{C}^{\rm min}$: Power-law scaling exponent for compute, approximately 0.050 · scalar