You are reading immutable version 3. The current guide may be newer.

Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resourcesā€”organized in teaching order.

All activities

Transformer Parameter and Compute Scaling

Transformer Forward Pass FLOPs Estimate: step by step

Interactive study view

Transformer Forward Pass FLOPs Estimate: step by step

Illustrative teaching data

Follow the existing bounded walkthrough in its intended sequence.

Step 1 of 3

  1. Step 1

    Calculate parameter-dependent FLOPs: 2 * N = 200,000,000

  2. Step 2

    Calculate attention-dependent FLOPs: 2 * n_layer * n_ctx * d_model = 37,748,736

  3. Step 3

    Sum both terms to get total forward FLOPs per token: 237,748,736