Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Contradictions and Conjectures at Scale

Scaling Contradictions and the Entropy Conjecture

Contradictions and Conjectures at Scale

Sources

section

6.3 Contradictions and a Conjecture

As we project scaling laws to extreme scales, we encounter an apparent contradiction between compute-efficient training and data limits. Extrapolating the optimal scaling trends suggests an intersection point where the optimal training parameters and data requirements reach a critical threshold:

Sources

S6.E8

C∗∼104PF-DaysN∗∼1012parameters,D∗∼1012tokens,L∗∼1.7nats/tokenformulae-sequencesimilar-tosuperscript𝐶superscript104PF-Daysformulae-sequencesimilar-tosuperscript𝑁superscript1012parametersformulae-sequencesimilar-tosuperscript𝐷superscript1012tokenssimilar-tosuperscript𝐿1.7nats/tokenC^{*}\sim 10^{4}\leavevmode\nobreak\ \mathrm{PF}{\text{-}}\mathrm{Days}\quad N^{*}\sim 10^{12}\leavevmode\nobreak\ \text{parameters},\quad D^{*}\sim 10^{12}\leavevmode\nobreak\ \text{tokens},\quad L^{*}\sim 1.7\leavevmode\nobreak\ \text{nats/token} (6.8)

C^{*}\sim 10^{4}\leavevmode\nobreak\ \mathrm{PF}{\text{-}}\mathrm{Days}\quad N^{*}\sim 10^{12}\leavevmode\nobreak\ \text{parameters},\quad D^{*}\sim 10^{12}\leavevmode\nobreak\ \text{tokens},\quad L^{*}\sim 1.7\leavevmode\nobreak\ \text{nats/token}

$C^{*}\sim 10^{4}\nobreak\ \mathrm{PF}{\text{-}}\mathrm{Days}\quad N^{*}\sim 10^{12}\nobreak\ \text{parameters},\quad D^{*}\sim 10^{12}\nobreak\ \text{tokens},\quad L^{*}\sim 1.7\nobreak\ \text{nats/token}$

Sources

S6.E8

C∗∼104PF-DaysN∗∼1012parameters,D∗∼1012tokens,L∗∼1.7nats/tokenformulae-sequencesimilar-tosuperscript𝐶superscript104PF-Daysformulae-sequencesimilar-tosuperscript𝑁superscript1012parametersformulae-sequencesimilar-tosuperscript𝐷superscript1012tokenssimilar-tosuperscript𝐿1.7nats/tokenC^{*}\sim 10^{4}\leavevmode\nobreak\ \mathrm{PF}{\text{-}}\mathrm{Days}\quad N^{*}\sim 10^{12}\leavevmode\nobreak\ \text{parameters},\quad D^{*}\sim 10^{12}\leavevmode\nobreak\ \text{tokens},\quad L^{*}\sim 1.7\leavevmode\nobreak\ \text{nats/token} (6.8)

C^{*}\sim 10^{4}\leavevmode\nobreak\ \mathrm{PF}{\text{-}}\mathrm{Days}\quad N^{*}\sim 10^{12}\leavevmode\nobreak\ \text{parameters},\quad D^{*}\sim 10^{12}\leavevmode\nobreak\ \text{tokens},\quad L^{*}\sim 1.7\leavevmode\nobreak\ \text{nats/token}

This intersection point suggests a physical or informational limit. If we cannot scale the model size beyond $N^*$ without encountering qualitatively different data requirements, it may imply that at this scale, we have extracted all of the reliable, learnable information available in natural language data. Under this conjecture, the loss $L^* \sim 1.7$ nats/token represents a fundamental limit: the approximate entropy-per-token of natural language. Beyond this point, we expect the cross-entropy loss trend to level off as models hit the true informational entropy of the medium.

Sources

S6.SS3.p8.5

One might also conjecture that this intersection point has a deeper meaning. If we cannot increase the model size beyond N∗superscript𝑁N^{*} without qualitatively different data requirements, perhaps this means that once we reach Cmin∗superscriptsubscript𝐶minC_{\rm min}^{*} and N∗superscript𝑁N^{*}, we have extracted all of the reliable information available in natural language data. In this interpretation, L∗superscript𝐿L^{*} would provide a rough estimate for the entropy-per-token666Defining words using the wc utility, the WebText2 dataset has 1.41.41.4 tokens per word and 4.34.34.3 characters per token. of natural language. In this scenario, we would expect the loss trend to level off at or before L∗superscript𝐿L^{*}.