You are reading immutable version 3. The current guide may be newer.

Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resourcesā€”organized in teaching order.

All activities

Transformer Parameter and Compute Scaling

Transformer Parameter Count Approximation: step by step

Interactive study view

Transformer Parameter Count Approximation: step by step

Illustrative teaching data

Follow the existing bounded walkthrough in its intended sequence.

Step 1 of 3

  1. Step 1

    Square the model dimension: 768^2 = 589,824

  2. Step 2

    Multiply by the number of layers: 12 * 589,824 = 7,077,888

  3. Step 3

    Multiply by the scaling factor 12: 12 * 7,077,888 = 84,934,656