Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Extreme Scaling: Contradictions and Conjectures

The Contradiction in Scaling Laws at Extreme Scales

Sources

S6.F15

Figure 15: Far beyond the model sizes we study empirically, we find a contradiction between our equations for L(Cmin)𝐿subscript𝐶minL(C_{\rm min}) and L(D)𝐿𝐷L(D) due to the slow growth of data needed for compute-efficient training. The intersection marks the point before which we expect our predictions to break down. The location of this point is highly sensitive to the precise exponents from our power-law fits.

When extrapolating scaling laws far beyond the empirically studied regime, a fundamental contradiction emerges between the equations governing compute-efficient training $L(C_{\rm min})$ and those governing data-limited training $L(D)$ . This contradiction arises because the amount of data required for compute-efficient training grows extremely slowly compared to the model size. At a certain extreme scale, the predicted optimal dataset size becomes smaller than what is required to avoid severe overfitting, causing the two scaling trends to intersect and break down.

Sources

S6.F15

Figure 15: Far beyond the model sizes we study empirically, we find a contradiction between our equations for L(Cmin)𝐿subscript𝐶minL(C_{\rm min}) and L(D)𝐿𝐷L(D) due to the slow growth of data needed for compute-efficient training. The intersection marks the point before which we expect our predictions to break down. The location of this point is highly sensitive to the precise exponents from our power-law fits.

Deep dive

$C^{*}\sim 10^{4}\nobreak\ \mathrm{PF}{\text{-}}\mathrm{Days}\quad N^{*}\sim 10^{12}\nobreak\ \text{parameters},\quad D^{*}\sim 10^{12}\nobreak\ \text{tokens},\quad L^{*}\sim 1.7\nobreak\ \text{nats/token}$

Sources

S6.E8

C∗∼104PF-DaysN∗∼1012parameters,D∗∼1012tokens,L∗∼1.7nats/tokenformulae-sequencesimilar-tosuperscript𝐶superscript104PF-Daysformulae-sequencesimilar-tosuperscript𝑁superscript1012parametersformulae-sequencesimilar-tosuperscript𝐷superscript1012tokenssimilar-tosuperscript𝐿1.7nats/tokenC^{*}\sim 10^{4}\leavevmode\nobreak\ \mathrm{PF}{\text{-}}\mathrm{Days}\quad N^{*}\sim 10^{12}\leavevmode\nobreak\ \text{parameters},\quad D^{*}\sim 10^{12}\leavevmode\nobreak\ \text{tokens},\quad L^{*}\sim 1.7\leavevmode\nobreak\ \text{nats/token} (6.8)

C^{*}\sim 10^{4}\leavevmode\nobreak\ \mathrm{PF}{\text{-}}\mathrm{Days}\quad N^{*}\sim 10^{12}\leavevmode\nobreak\ \text{parameters},\quad D^{*}\sim 10^{12}\leavevmode\nobreak\ \text{tokens},\quad L^{*}\sim 1.7\leavevmode\nobreak\ \text{nats/token}

Deep dive

This intersection point may carry a deeper physical meaning. If we cannot scale the model size beyond $N^*$ parameters without qualitatively different data requirements, it suggests that at this scale, we have extracted all the reliable, learnable information available in natural language. Under this interpretation, the loss value $L^* \sim 1.7$ nats/token represents a rough estimate of the fundamental entropy-per-token of natural language, where the loss trend is expected to level off.

Sources

S6.SS3.p8.5

One might also conjecture that this intersection point has a deeper meaning. If we cannot increase the model size beyond N∗superscript𝑁N^{*} without qualitatively different data requirements, perhaps this means that once we reach Cmin∗superscriptsubscript𝐶minC_{\rm min}^{*} and N∗superscript𝑁N^{*}, we have extracted all of the reliable information available in natural language data. In this interpretation, L∗superscript𝐿L^{*} would provide a rough estimate for the entropy-per-token666Defining words using the wc utility, the WebText2 dataset has 1.41.41.4 tokens per word and 4.34.34.3 characters per token. of natural language. In this scenario, we would expect the loss trend to level off at or before L∗superscript𝐿L^{*}.