You are reading immutable version 3. The current guide may be newer.

Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resourcesā€”organized in teaching order.

All activities

Transformer Parameter and Compute Scaling

Optimal Compute Allocation Scaling Relations: step by step

Interactive study view

Optimal Compute Allocation Scaling Relations: step by step

Illustrative teaching data

Follow the existing bounded walkthrough in its intended sequence.

Step 1 of 7

  1. Step 1

    Calculate the exponent for N: 0.042 / 0.076 = 0.5526

  2. Step 2

    Calculate the relative scaling factor for N: 100^0.5526 = 12.74

  3. Step 3

    Calculate the exponent for B: 0.042 / 0.21 = 0.2000

  4. Step 4

    Calculate the relative scaling factor for B: 100^0.20 = 2.512

  5. Step 5

    Calculate the exponent for S: 0.042 / 0.17 = 0.2471

  6. Step 6

    Calculate the relative scaling factor for S: 100^0.2471 = 3.120

  7. Step 7

    Calculate the relative scaling factor for D: B * S = 2.512 * 3.120 = 7.837