Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Overview of Scaling Laws for Language Models

At a glance

Scaling Laws for Neural Language Models

Sources

section

1 Introduction

Language modeling performance improves smoothly and predictably when scaling up three primary factors: model size (number of non-embedding parameters $N$ ), dataset size ( $D$ ), and the amount of compute used for training ( $C$ ). When performance is not bottlenecked by the other two factors, the cross-entropy loss $L$ exhibits a power-law relationship with each individual factor.

Sources

S1.F1

Figure 1: Language modeling performance improves smoothly as we increase the model size, datasetset size, and amount of compute111Here we display predicted compute when using a sufficiently small batch size. See Figure 13 for comparison to the purely empirical data. used for training. For optimal performance all three factors must be scaled up in tandem. Empirical performance has a power-law relationship with each individual factor when not bottlenecked by the other two.

Specifically, the scaling of the loss with respect to each of these parameters is described by the following power-law relations:

Sources

S1.E1

L(N)=(Nc/N)αN;αN∼0.076,Nc∼8.8×1013(non-embedding parameters)formulae-sequence𝐿𝑁superscriptsubscript𝑁c𝑁subscript𝛼𝑁formulae-sequencesimilar-tosubscript𝛼𝑁0.076similar-tosubscript𝑁c8.8superscript1013(non-embedding parameters)L(N)=\left(N_{\mathrm{c}}/N\right)^{\alpha_{N}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{N}\sim 0.076,\quad N_{\mathrm{c}}\sim 8.8\times 10^{13}\leavevmode\nobreak\ \text{(non-embedding parameters)} (1.1)

L(N)=\left(N_{\mathrm{c}}/N\right)^{\alpha_{N}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{N}\sim 0.076,\quad N_{\mathrm{c}}\sim 8.8\times 10^{13}\leavevmode\nobreak\ \text{(non-embedding parameters)}

S1.E2

L(D)=(Dc/D)αD;αD∼0.095,Dc∼5.4×1013(tokens)formulae-sequence𝐿𝐷superscriptsubscript𝐷c𝐷subscript𝛼𝐷formulae-sequencesimilar-tosubscript𝛼𝐷0.095similar-tosubscript𝐷c5.4superscript1013(tokens)L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\leavevmode\nobreak\ \text{(tokens)} (1.2)

L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\leavevmode\nobreak\ \text{(tokens)}

S1.E3

L(Cmin)=(Ccmin/Cmin)αCmin;αCmin∼0.050,Ccmin∼3.1×108(PF-days)formulae-sequence𝐿subscript𝐶minsuperscriptsuperscriptsubscript𝐶cminsubscript𝐶minsuperscriptsubscript𝛼𝐶minformulae-sequencesimilar-tosuperscriptsubscript𝛼𝐶min0.050similar-tosuperscriptsubscript𝐶cmin3.1superscript108(PF-days)L(C_{\rm min})=\left(C_{\mathrm{c}}^{\rm min}/C_{\rm min}\right)^{\alpha_{C}^{\rm min}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{C}^{\rm min}\sim 0.050,\quad C_{\mathrm{c}}^{\rm min}\sim 3.1\times 10^{8}\leavevmode\nobreak\ \text{(PF-days)} (1.3)

L(C_{\rm min})=\left(C_{\mathrm{c}}^{\rm min}/C_{\rm min}\right)^{\alpha_{C}^{\rm min}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{C}^{\rm min}\sim 0.050,\quad C_{\mathrm{c}}^{\rm min}\sim 3.1\times 10^{8}\leavevmode\nobreak\ \text{(PF-days)}

$L(N)=\left(N_{\mathrm{c}}/N\right)^{\alpha_{N}};\nobreak\ \nobreak\ \alpha_{N}\sim 0.076,\quad N_{\mathrm{c}}\sim 8.8\times 10^{13}\nobreak\ \text{(non-embedding parameters)}$

Sources

S1.E1

L(N)=(Nc/N)αN;αN∼0.076,Nc∼8.8×1013(non-embedding parameters)formulae-sequence𝐿𝑁superscriptsubscript𝑁c𝑁subscript𝛼𝑁formulae-sequencesimilar-tosubscript𝛼𝑁0.076similar-tosubscript𝑁c8.8superscript1013(non-embedding parameters)L(N)=\left(N_{\mathrm{c}}/N\right)^{\alpha_{N}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{N}\sim 0.076,\quad N_{\mathrm{c}}\sim 8.8\times 10^{13}\leavevmode\nobreak\ \text{(non-embedding parameters)} (1.1)

L(N)=\left(N_{\mathrm{c}}/N\right)^{\alpha_{N}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{N}\sim 0.076,\quad N_{\mathrm{c}}\sim 8.8\times 10^{13}\leavevmode\nobreak\ \text{(non-embedding parameters)}

$L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\nobreak\ \nobreak\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\nobreak\ \text{(tokens)}$

Sources

S1.E2

L(D)=(Dc/D)αD;αD∼0.095,Dc∼5.4×1013(tokens)formulae-sequence𝐿𝐷superscriptsubscript𝐷c𝐷subscript𝛼𝐷formulae-sequencesimilar-tosubscript𝛼𝐷0.095similar-tosubscript𝐷c5.4superscript1013(tokens)L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\leavevmode\nobreak\ \text{(tokens)} (1.2)

L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\leavevmode\nobreak\ \text{(tokens)}

$L(C_{\rm min})=\left(C_{\mathrm{c}}^{\rm min}/C_{\rm min}\right)^{\alpha_{C}^{\rm min}};\nobreak\ \nobreak\ \alpha_{C}^{\rm min}\sim 0.050,\quad C_{\mathrm{c}}^{\rm min}\sim 3.1\times 10^{8}\nobreak\ \text{(PF-days)}$

Sources

S1.E3

L(Cmin)=(Ccmin/Cmin)αCmin;αCmin∼0.050,Ccmin∼3.1×108(PF-days)formulae-sequence𝐿subscript𝐶minsuperscriptsuperscriptsubscript𝐶cminsubscript𝐶minsuperscriptsubscript𝛼𝐶minformulae-sequencesimilar-tosuperscriptsubscript𝛼𝐶min0.050similar-tosuperscriptsubscript𝐶cmin3.1superscript108(PF-days)L(C_{\rm min})=\left(C_{\mathrm{c}}^{\rm min}/C_{\rm min}\right)^{\alpha_{C}^{\rm min}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{C}^{\rm min}\sim 0.050,\quad C_{\mathrm{c}}^{\rm min}\sim 3.1\times 10^{8}\leavevmode\nobreak\ \text{(PF-days)} (1.3)

L(C_{\rm min})=\left(C_{\mathrm{c}}^{\rm min}/C_{\rm min}\right)^{\alpha_{C}^{\rm min}};\leavevmode\nobreak\ \leavevmode\nobreak\ \alpha_{C}^{\rm min}\sim 0.050,\quad C_{\mathrm{c}}^{\rm min}\sim 3.1\times 10^{8}\leavevmode\nobreak\ \text{(PF-days)}