You are reading immutable version 3. The current guide may be newer.

Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

The Infinite Data Limit and Overfitting

The Infinite Data Limit and Overfitting

The Infinite Data Limit and Overfitting

Joint Scaling and Overfitting

Sources

S1.E5

L​(N,D)=[(NcN)αNαD+DcD]αD𝐿𝑁𝐷superscriptdelimited-[]superscriptsubscript𝑁𝑐𝑁subscript𝛼𝑁subscript𝛼𝐷subscript𝐷𝑐𝐷subscript𝛼𝐷L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} (1.5)
L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}

To capture the simultaneous dependence of the cross-entropy loss on both the model size NN and the dataset size DD, we can model the loss using a unified scaling law. This formulation parameterizes how performance degrades when the dataset size is finite, leading to overfitting.

Sources

S1.E5

L​(N,D)=[(NcN)αNαD+DcD]αD𝐿𝑁𝐷superscriptdelimited-[]superscriptsubscript𝑁𝑐𝑁subscript𝛼𝑁subscript𝛼𝐷subscript𝐷𝑐𝐷subscript𝛼𝐷L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} (1.5)
L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}

$$L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}$$$$

Sources

S1.E5

L​(N,D)=[(NcN)αNαD+DcD]αD𝐿𝑁𝐷superscriptdelimited-[]superscriptsubscript𝑁𝑐𝑁subscript𝛼𝑁subscript𝛼𝐷subscript𝐷𝑐𝐷subscript𝛼𝐷L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} (1.5)
L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}

To avoid significant performance degradation due to overfitting, the dataset size must scale with the model size. Specifically, the relationship between the dataset size DD and model parameter count NN to prevent bottlenecking is given by:

Sources

S4.E4

D≳(5×103)​N0.74greater-than-or-equivalent-to𝐷5superscript103superscript𝑁0.74D\gtrsim(5\times 10^{3})\,N^{0.74} (4.4)
D\gtrsim(5\times 10^{3})\,N^{0.74}

D≳(5×103) N0.74D\gtrsim(5\times 10^{3})\,N^{0.74}

Sources

S4.E4

D≳(5×103)​N0.74greater-than-or-equivalent-to𝐷5superscript103superscript𝑁0.74D\gtrsim(5\times 10^{3})\,N^{0.74} (4.4)
D\gtrsim(5\times 10^{3})\,N^{0.74}