You are reading immutable version 4. The current guide may be newer.

Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Overfitting and the Infinite Data Limit

Optimal Allocation Scaling: step by step

Interactive study view

Optimal Allocation Scaling: step by step

Illustrative teaching data

Follow the existing bounded walkthrough in its intended sequence.

Step 1 of 3

  1. Step 1

    Calculate the exponent for model size N: alpha_C_min / alpha_N = 0.042 / 0.076 = 0.5526

  2. Step 2

    Calculate the exponent for batch size B: alpha_C_min / alpha_B = 0.042 / 0.21 = 0.2000

  3. Step 3

    Calculate the exponent for training steps S: alpha_C_min / alpha_S = 0.042 / 0.17 = 0.2471