You are reading immutable version 3. The current guide may be newer.

Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resourcesā€”organized in teaching order.

All activities

Critical Batch Size and Training Dynamics

Critical Batch Size Power Law: step by step

Interactive study view

Critical Batch Size Power Law: step by step

Illustrative teaching data

Follow the existing bounded walkthrough in its intended sequence.

Step 1 of 3

  1. Step 1

    Calculate the exponent: 1 / 0.21 = 4.7619

  2. Step 2

    Calculate the denominator: 2.0^4.7619 = 27.135

  3. Step 3

    Divide B_* by the denominator: 200,000,000 / 27.135 = 7,370,554.6