Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Optimal Allocation of Compute Budgets

Intersection point of scaling laws

The intersection point where the scaling laws suggest a potential contradiction or transition is characterized by a compute budget $C^* \sim 10^4$ PF-Days, model size $N^* \sim 10^{12}$ parameters, dataset size $D^* \sim 10^{12}$ tokens, and loss $L^* \sim 1.7$ nats/token.

Sources

S6.E8

C∗∼104PF-DaysN∗∼1012parameters,D∗∼1012tokens,L∗∼1.7nats/tokenformulae-sequencesimilar-tosuperscript𝐶superscript104PF-Daysformulae-sequencesimilar-tosuperscript𝑁superscript1012parametersformulae-sequencesimilar-tosuperscript𝐷superscript1012tokenssimilar-tosuperscript𝐿1.7nats/tokenC^{*}\sim 10^{4}\leavevmode\nobreak\ \mathrm{PF}{\text{-}}\mathrm{Days}\quad N^{*}\sim 10^{12}\leavevmode\nobreak\ \text{parameters},\quad D^{*}\sim 10^{12}\leavevmode\nobreak\ \text{tokens},\quad L^{*}\sim 1.7\leavevmode\nobreak\ \text{nats/token} (6.8)

C^{*}\sim 10^{4}\leavevmode\nobreak\ \mathrm{PF}{\text{-}}\mathrm{Days}\quad N^{*}\sim 10^{12}\leavevmode\nobreak\ \text{parameters},\quad D^{*}\sim 10^{12}\leavevmode\nobreak\ \text{tokens},\quad L^{*}\sim 1.7\leavevmode\nobreak\ \text{nats/token}

Reported values
Measure	Value

Sources

S6.E8

C∗∼104PF-DaysN∗∼1012parameters,D∗∼1012tokens,L∗∼1.7nats/tokenformulae-sequencesimilar-tosuperscript𝐶superscript104PF-Daysformulae-sequencesimilar-tosuperscript𝑁superscript1012parametersformulae-sequencesimilar-tosuperscript𝐷superscript1012tokenssimilar-tosuperscript𝐿1.7nats/tokenC^{*}\sim 10^{4}\leavevmode\nobreak\ \mathrm{PF}{\text{-}}\mathrm{Days}\quad N^{*}\sim 10^{12}\leavevmode\nobreak\ \text{parameters},\quad D^{*}\sim 10^{12}\leavevmode\nobreak\ \text{tokens},\quad L^{*}\sim 1.7\leavevmode\nobreak\ \text{nats/token} (6.8)

C^{*}\sim 10^{4}\leavevmode\nobreak\ \mathrm{PF}{\text{-}}\mathrm{Days}\quad N^{*}\sim 10^{12}\leavevmode\nobreak\ \text{parameters},\quad D^{*}\sim 10^{12}\leavevmode\nobreak\ \text{tokens},\quad L^{*}\sim 1.7\leavevmode\nobreak\ \text{nats/token}