You are reading immutable version 4. The current guide may be newer.

Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Critical Batch Size and Training Dynamics

Critical Batch Size Power Law

Critical Batch Size Power Law

Source equation

Bcrit(L)=B∗L1/αB,B∗∼2⋅108 tokens,  αB∼0.21B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21

The critical batch size Bcrit(L)B_{\rm crit}(L) determines the optimal batch size for training neural language models. It represents the threshold where increasing the batch size further yields diminishing returns in terms of training speedup per unit of compute. This relationship is modeled as a power law of the cross-entropy loss LL.

Sources

S1.E4

Bcrit​(L)=B∗L1/αB,B∗∼2⋅108​ tokens,αB∼0.21formulae-sequencesubscript𝐵crit𝐿subscript𝐵∗superscript𝐿1subscript𝛼𝐵formulae-sequencesimilar-tosubscript𝐵∗⋅2superscript108 tokenssimilar-tosubscript𝛼𝐵0.21B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21 (1.4)
B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21

Bcrit(L)=B∗L1/αB,B∗∼2⋅108 tokens,  αB∼0.21B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21

Sources

S1.E4

Bcrit​(L)=B∗L1/αB,B∗∼2⋅108​ tokens,αB∼0.21formulae-sequencesubscript𝐵crit𝐿subscript𝐵∗superscript𝐿1subscript𝛼𝐵formulae-sequencesimilar-tosubscript𝐵∗⋅2superscript108 tokenssimilar-tosubscript𝛼𝐵0.21B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21 (1.4)
B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21

Illustrative Calculation

Let us calculate the critical batch size Bcrit(L)B_{\rm crit}(L) for a target cross-entropy loss of L=2.0L = 2.0:

  1. Given Parameters:

    • B∗=2⋅108B_{\ast} = 2 \cdot 10^8 tokens
    • αB=0.21\alpha_B = 0.21
    • L=2.0L = 2.0

  2. Compute the Exponent:

    • 1/αB=1/0.21≈4.76191 / \alpha_B = 1 / 0.21 \approx 4.7619

  3. Compute the Loss Term:

    • L1/αB=2.04.7619≈27.135L^{1/\alpha_B} = 2.0^{4.7619} \approx 27.135

  4. Calculate BcritB_{\rm crit}:

    • Bcrit(2.0)=2⋅10827.135≈7.37⋅106B_{\rm crit}(2.0) = \frac{2 \cdot 10^8}{27.135} \approx 7.37 \cdot 10^6 tokens
Sources

S1.E4

Bcrit​(L)=B∗L1/αB,B∗∼2⋅108​ tokens,αB∼0.21formulae-sequencesubscript𝐵crit𝐿subscript𝐵∗superscript𝐿1subscript𝛼𝐵formulae-sequencesimilar-tosubscript𝐵∗⋅2superscript108 tokenssimilar-tosubscript𝛼𝐵0.21B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21 (1.4)
B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21
Bcrit(L)B_{\rm crit}(L)
Critical batch size measured in tokens · scalar
LL
Cross-entropy loss value · scalar
B∗B_{\ast}
Scale parameter for the critical batch size · scalar
αB\alpha_{B}
Power-law scaling exponent for the critical batch size · scalar