You are reading immutable version 1. The current guide may be newer.

Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Critical Batch Size and Training Dynamics

Critical batch size power law

Critical batch size power law

Source equation

Bcrit(L)=B∗L1/αB,B∗∼2⋅108 tokens,  αB∼0.21B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21

The critical batch size Bcrit(L)B_{\rm crit}(L) determines the optimal batch size for training language models. It represents the point of diminishing returns where increasing the batch size further yields minimal parallelization speedup per unit of compute. This relationship is modeled as a power law of the cross-entropy loss LL.

Sources

S1.E4

Bcrit​(L)=B∗L1/αB,B∗∼2⋅108​ tokens,αB∼0.21formulae-sequencesubscript𝐵crit𝐿subscript𝐵∗superscript𝐿1subscript𝛼𝐵formulae-sequencesimilar-tosubscript𝐵∗⋅2superscript108 tokenssimilar-tosubscript𝛼𝐵0.21B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21 (1.4)
B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21

Bcrit(L)=B∗L1/αB,B∗∼2⋅108 tokens,  αB∼0.21B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21

Sources

S1.E4

Bcrit​(L)=B∗L1/αB,B∗∼2⋅108​ tokens,αB∼0.21formulae-sequencesubscript𝐵crit𝐿subscript𝐵∗superscript𝐿1subscript𝛼𝐵formulae-sequencesimilar-tosubscript𝐵∗⋅2superscript108 tokenssimilar-tosubscript𝛼𝐵0.21B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21 (1.4)
B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21

Illustrative Calculation

Let us calculate the critical batch size Bcrit(L)B_{\rm crit}(L) for a target cross-entropy loss of L=2.0L = 2.0.

Using the empirical parameters:

  • B∗=2×108B_{\ast} = 2 \times 10^8 tokens
  • αB=0.21\alpha_B = 0.21

  1. Calculate the exponent: 1αB=10.21≈4.7619\frac{1}{\alpha_B} = \frac{1}{0.21} \approx 4.7619

  2. Compute the loss term: L1/αB=2.04.7619≈27.135L^{1/\alpha_B} = 2.0^{4.7619} \approx 27.135

  3. Calculate Bcrit(L)B_{\rm crit}(L): Bcrit(2.0)=2×10827.135≈7.37×106 tokensB_{\rm crit}(2.0) = \frac{2 \times 10^8}{27.135} \approx 7.37 \times 10^6 \text{ tokens}

This demonstrates that as the loss LL decreases (i.e., the model becomes more trained and accurate), the critical batch size Bcrit(L)B_{\rm crit}(L) increases significantly.

Sources

S1.E4

Bcrit​(L)=B∗L1/αB,B∗∼2⋅108​ tokens,αB∼0.21formulae-sequencesubscript𝐵crit𝐿subscript𝐵∗superscript𝐿1subscript𝛼𝐵formulae-sequencesimilar-tosubscript𝐵∗⋅2superscript108 tokenssimilar-tosubscript𝛼𝐵0.21B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21 (1.4)
B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21
Bcrit(L)B_{\rm crit}(L)
Critical batch size measured in tokens · scalar
LL
Cross-entropy loss · scalar
B∗B_{\ast}
Scale parameter for the critical batch size · scalar
αB\alpha_{B}
Power-law scaling exponent for the critical batch size · scalar