You are reading immutable version 3. The current guide may be newer.

Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Transformer Parameter and Compute Scaling

Transformer Forward Pass FLOPs Estimate

Transformer Forward Pass FLOPs Estimate

Source equation

Cforward≈2N+2nlayernctxdmodelC_{\rm forward}\approx 2N+2n_{{\rm layer}}n_{{\rm ctx}}d_{{\rm model}}

This equation estimates the computational cost (in FLOPs per token) of a forward pass through a standard Transformer model. It accounts for both the parameter-dependent operations (such as feed-forward networks and projections) and the context-dependent attention operations.

Sources

equation

Cforward≈2​N+2​nlayer​nctx​dmodelsubscript𝐶forward2𝑁2subscript𝑛layersubscript𝑛ctxsubscript𝑑modelC_{\rm forward}\approx 2N+2n_{{\rm layer}}n_{{\rm ctx}}d_{{\rm model}} (2.2)
C_{\rm forward}\approx 2N+2n_{{\rm layer}}n_{{\rm ctx}}d_{{\rm model}}

Cforward≈2N+2nlayernctxdmodelC_{\rm forward}\approx 2N+2n_{{\rm layer}}n_{{\rm ctx}}d_{{\rm model}}

Sources

equation

Cforward≈2​N+2​nlayer​nctx​dmodelsubscript𝐶forward2𝑁2subscript𝑛layersubscript𝑛ctxsubscript𝑑modelC_{\rm forward}\approx 2N+2n_{{\rm layer}}n_{{\rm ctx}}d_{{\rm model}} (2.2)
C_{\rm forward}\approx 2N+2n_{{\rm layer}}n_{{\rm ctx}}d_{{\rm model}}

The term 2N2N represents the FLOPs required for the matrix multiplications associated with the non-embedding parameters NN (since each parameter is used in a multiply-accumulate operation, which counts as 2 FLOPs per token). The term 2nlayernctxdmodel2n_{\rm layer}n_{\rm ctx}d_{\rm model} represents the FLOPs required for the attention matrix multiplications (query-key dot products and attention-value multiplication) which scale quadratically with the context length nctxn_{\rm ctx}.

Sources

equation

Cforward≈2​N+2​nlayer​nctx​dmodelsubscript𝐶forward2𝑁2subscript𝑛layersubscript𝑛ctxsubscript𝑑modelC_{\rm forward}\approx 2N+2n_{{\rm layer}}n_{{\rm ctx}}d_{{\rm model}} (2.2)
C_{\rm forward}\approx 2N+2n_{{\rm layer}}n_{{\rm ctx}}d_{{\rm model}}
Implementation detail

Illustrative Calculation

Let's calculate the forward pass FLOPs per token for a small Transformer model with the following parameters:

  • Non-embedding parameters N=100,000,000N = 100,000,000 (100M)
  • Number of layers nlayer=12n_{\rm layer} = 12
  • Context length nctx=2048n_{\rm ctx} = 2048
  • Model dimension dmodel=768d_{\rm model} = 768

Using the formula:

  1. Calculate the parameter-dependent term: 2N=2×100,000,000=200,000,000 FLOPs2N = 2 \times 100,000,000 = 200,000,000\text{ FLOPs}
  2. Calculate the attention-dependent term: 2nlayernctxdmodel=2×12×2048×768=37,748,736 FLOPs2n_{\rm layer}n_{\rm ctx}d_{\rm model} = 2 \times 12 \times 2048 \times 768 = 37,748,736\text{ FLOPs}
  3. Sum the terms: Cforward≈200,000,000+37,748,736=237,748,736 FLOPs per tokenC_{\rm forward} \approx 200,000,000 + 37,748,736 = 237,748,736\text{ FLOPs per token}
Sources

equation

Cforward≈2​N+2​nlayer​nctx​dmodelsubscript𝐶forward2𝑁2subscript𝑛layersubscript𝑛ctxsubscript𝑑modelC_{\rm forward}\approx 2N+2n_{{\rm layer}}n_{{\rm ctx}}d_{{\rm model}} (2.2)
C_{\rm forward}\approx 2N+2n_{{\rm layer}}n_{{\rm ctx}}d_{{\rm model}}
CforwardC_{\rm forward}
Computational cost of a forward pass per token (in FLOPs) · scalar
NN
Number of non-embedding parameters in the model · scalar
nlayern_{\rm layer}
Number of layers in the Transformer · scalar
nctxn_{\rm ctx}
Context window length (number of tokens) · scalar
dmodeld_{\rm model}
Hidden dimension size of the model · scalar