Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Basic Power Laws of Scaling

Critical batch size as a power law of loss

Source equation

$B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21$

The critical batch size $B_{\rm crit}(L)$ determines the boundary where increasing the batch size further yields diminishing returns in terms of optimization speedup per step. This relationship is modeled as a power law of the cross-entropy loss $L$ .

Sources

S1.E4

Bcrit(L)=B∗L1/αB,B∗∼2⋅108 tokens,αB∼0.21formulae-sequencesubscript𝐵crit𝐿subscript𝐵∗superscript𝐿1subscript𝛼𝐵formulae-sequencesimilar-tosubscript𝐵∗⋅2superscript108 tokenssimilar-tosubscript𝛼𝐵0.21B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21 (1.4)

B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21

$B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21$

Sources

S1.E4

Bcrit(L)=B∗L1/αB,B∗∼2⋅108 tokens,αB∼0.21formulae-sequencesubscript𝐵crit𝐿subscript𝐵∗superscript𝐿1subscript𝛼𝐵formulae-sequencesimilar-tosubscript𝐵∗⋅2superscript108 tokenssimilar-tosubscript𝛼𝐵0.21B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21 (1.4)

B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21

As the loss $L$ decreases during training, the critical batch size $B_{\rm crit}(L)$ increases. This implies that larger models or models trained to a lower loss can effectively utilize much larger batch sizes without experiencing parallelization bottlenecks.

Sources

S1.E4

Bcrit(L)=B∗L1/αB,B∗∼2⋅108 tokens,αB∼0.21formulae-sequencesubscript𝐵crit𝐿subscript𝐵∗superscript𝐿1subscript𝛼𝐵formulae-sequencesimilar-tosubscript𝐵∗⋅2superscript108 tokenssimilar-tosubscript𝛼𝐵0.21B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21 (1.4)

B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21

Implementation detail

Illustrative Calculation

Let us calculate the critical batch size $B_{\rm crit}(L)$ for a target loss value of $L = 2.0$ :

Inputs:
- $L = 2.0$
- $B_{\ast} = 2 \cdot 10^8$ tokens
- $\alpha_B = 0.21$
Step-by-step Evaluation:
- Calculate the exponent: $1 / \alpha_B = 1 / 0.21 \approx 4.7619$
- Compute the denominator: $L^{1/\alpha_B} = 2.0^{4.7619} \approx 27.131$
- Compute the critical batch size: $B_{\rm crit}(2.0) = \frac{2 \cdot 10^8}{27.131} \approx 7.37 \cdot 10^6$ tokens

This indicates that at a loss of $2.0$ , the critical batch size is approximately $7.37$ million tokens.

Sources

S1.E4

Bcrit(L)=B∗L1/αB,B∗∼2⋅108 tokens,αB∼0.21formulae-sequencesubscript𝐵crit𝐿subscript𝐵∗superscript𝐿1subscript𝛼𝐵formulae-sequencesimilar-tosubscript𝐵∗⋅2superscript108 tokenssimilar-tosubscript𝛼𝐵0.21B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21 (1.4)

B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21

$B_{\rm crit}(L)$: Critical batch size as a function of loss · scalar
$L$: Cross-entropy loss value · scalar
$B_{\ast}$: Scale parameter for the critical batch size · scalar
$\alpha_{B}$: Power-law scaling exponent for the critical batch size · scalar