Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Critical Batch Size and Training Dynamics

Critical Batch Size Power Law: step by step

Interactive study view

Critical Batch Size Power Law: step by step

Illustrative teaching data

Follow the existing bounded walkthrough in its intended sequence.

Step 1 of 4

  1. Step 1

    Identify the constant parameters: B_* = 2 * 10^8 and alpha_B = 0.21.

  2. Step 2

    Calculate the exponent: 1 / alpha_B = 1 / 0.21 ≈ 4.7619.

  3. Step 3

    Evaluate the denominator: L^(1 / alpha_B) = 2.0^4.7619 ≈ 27.135.

  4. Step 4

    Divide B_* by the denominator to find B_crit: 2 * 10^8 / 27.135 ≈ 7,370,554.6 tokens.