You are reading immutable version 2. The current guide may be newer.

Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Critical Batch Size and Training Dynamics

Critical Batch Size and Training Dynamics

Critical Batch Size and Training Dynamics

Critical Batch Size Scaling

Sources

section

5.1 Adjustment for Training at Bcrit​(L)subscript𝐵crit𝐿B_{\rm crit}(L)

The critical batch size BcritB_{\rm crit} determines the boundary between data-parallel efficiency and diminishing returns. It scales as a power-law of the cross-entropy loss LL:

Sources

S1.E4

Bcrit​(L)=B∗L1/αB,B∗∼2⋅108​ tokens,αB∼0.21formulae-sequencesubscript𝐵crit𝐿subscript𝐵∗superscript𝐿1subscript𝛼𝐵formulae-sequencesimilar-tosubscript𝐵∗⋅2superscript108 tokenssimilar-tosubscript𝛼𝐵0.21B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21 (1.4)
B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21

Bcrit(L)=B∗L1/αB,B∗∼2⋅108 tokens,  αB∼0.21B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21

Sources

S1.E4

Bcrit​(L)=B∗L1/αB,B∗∼2⋅108​ tokens,αB∼0.21formulae-sequencesubscript𝐵crit𝐿subscript𝐵∗superscript𝐿1subscript𝛼𝐵formulae-sequencesimilar-tosubscript𝐵∗⋅2superscript108 tokenssimilar-tosubscript𝛼𝐵0.21B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21 (1.4)
B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21

When training with a batch size BB, we can define the minimum number of steps SminS_{\rm min} and minimum compute CminC_{\rm min} required to reach a given loss level. These quantities represent the idealized limits when operating far from the bottlenecks:

Sources

S5.E4

Smin​(S)≡S1+Bcrit​(L)/B(minimum steps, at ​B≫Bcrit)subscript𝑆min𝑆𝑆1subscript𝐵crit𝐿𝐵much-greater-thanminimum steps, at 𝐵subscript𝐵critS_{\rm min}(S)\equiv\frac{S}{1+B_{\rm crit}(L)/B}\qquad(\text{minimum steps, at }B\gg B_{\rm crit}) (5.4)
S_{\rm min}(S)\equiv\frac{S}{1+B_{\rm crit}(L)/B}\qquad(\text{minimum steps, at }B\gg B_{\rm crit})

S5.E5

Cmin​(C)≡C1+B/Bcrit​(L)(minimum compute, at ​B≪Bcrit)subscript𝐶min𝐶𝐶1𝐵subscript𝐵crit𝐿much-less-thanminimum compute, at 𝐵subscript𝐵critC_{\rm min}(C)\equiv\frac{C}{1+B/B_{\rm crit}(L)}\qquad(\text{minimum compute, at }B\ll B_{\rm crit}) (5.5)
C_{\rm min}(C)\equiv\frac{C}{1+B/B_{\rm crit}(L)}\qquad(\text{minimum compute, at }B\ll B_{\rm crit})

Smin(S)≡S1+Bcrit(L)/B(minimum steps, at B≫Bcrit)S_{\rm min}(S)\equiv\frac{S}{1+B_{\rm crit}(L)/B}\qquad(\text{minimum steps, at }B\gg B_{\rm crit})

Sources

S5.E4

Smin​(S)≡S1+Bcrit​(L)/B(minimum steps, at ​B≫Bcrit)subscript𝑆min𝑆𝑆1subscript𝐵crit𝐿𝐵much-greater-thanminimum steps, at 𝐵subscript𝐵critS_{\rm min}(S)\equiv\frac{S}{1+B_{\rm crit}(L)/B}\qquad(\text{minimum steps, at }B\gg B_{\rm crit}) (5.4)
S_{\rm min}(S)\equiv\frac{S}{1+B_{\rm crit}(L)/B}\qquad(\text{minimum steps, at }B\gg B_{\rm crit})

Cmin(C)≡C1+B/Bcrit(L)(minimum compute, at B≪Bcrit)C_{\rm min}(C)\equiv\frac{C}{1+B/B_{\rm crit}(L)}\qquad(\text{minimum compute, at }B\ll B_{\rm crit})

Sources

S5.E5

Cmin​(C)≡C1+B/Bcrit​(L)(minimum compute, at ​B≪Bcrit)subscript𝐶min𝐶𝐶1𝐵subscript𝐵crit𝐿much-less-thanminimum compute, at 𝐵subscript𝐵critC_{\rm min}(C)\equiv\frac{C}{1+B/B_{\rm crit}(L)}\qquad(\text{minimum compute, at }B\ll B_{\rm crit}) (5.5)
C_{\rm min}(C)\equiv\frac{C}{1+B/B_{\rm crit}(L)}\qquad(\text{minimum compute, at }B\ll B_{\rm crit})