Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Transformer Parameter and Compute Scaling

Transformer Parameter Count Approximation

Source equation

$\displaystyle=12n_{\rm layer}d_{{\rm model}}^{2}\quad\text{ with the standard }\quad d_{\rm attn}=d_{\rm ff}/4=d_{{\rm model}}$

This equation estimates the total number of non-embedding parameters in a standard Transformer model. It assumes standard scaling relationships between the model dimension, attention dimension, and feed-forward dimension.

Sources

equation

=12nlayerdmodel2 with the standard dattn=dff/4=dmodelformulae-sequenceabsent12subscript𝑛layersuperscriptsubscript𝑑model2 with the standard subscript𝑑attnsubscript𝑑ff4subscript𝑑model\displaystyle=12n_{\rm layer}d_{{\rm model}}^{2}\quad\text{ with the standard }\quad d_{\rm attn}=d_{\rm ff}/4=d_{{\rm model}} (2.1)

\displaystyle=12n_{\rm layer}d_{{\rm model}}^{2}\quad\text{ with the standard }\quad d_{\rm attn}=d_{\rm ff}/4=d_{{\rm model}}

$\displaystyle=12n_{\rm layer}d_{{\rm model}}^{2}\quad\text{ with the standard }\quad d_{\rm attn}=d_{\rm ff}/4=d_{{\rm model}}$

Sources

equation

=12nlayerdmodel2 with the standard dattn=dff/4=dmodelformulae-sequenceabsent12subscript𝑛layersuperscriptsubscript𝑑model2 with the standard subscript𝑑attnsubscript𝑑ff4subscript𝑑model\displaystyle=12n_{\rm layer}d_{{\rm model}}^{2}\quad\text{ with the standard }\quad d_{\rm attn}=d_{\rm ff}/4=d_{{\rm model}} (2.1)

\displaystyle=12n_{\rm layer}d_{{\rm model}}^{2}\quad\text{ with the standard }\quad d_{\rm attn}=d_{\rm ff}/4=d_{{\rm model}}

Deep dive

For each Transformer layer, the parameters are divided into two main blocks:

Self-Attention Block: Contains projection matrices for Query, Key, Value, and Output. Each has shape $[d_{\rm model}, d_{\rm model}]$ , contributing $4 d_{\rm model}^2$ parameters.
Feed-Forward Network (FFN): Consists of two linear layers. The first projects from $d_{\rm model}$ to $d_{\rm ff} = 4d_{\rm model}$ , and the second projects back to $d_{\rm model}$ . This contributes $2 \times d_{\rm model} \times 4d_{\rm model} = 8 d_{\rm model}^2$ parameters.

Summing these gives $4 d_{\rm model}^2 + 8 d_{\rm model}^2 = 12 d_{\rm model}^2$ parameters per layer. Multiplying by the number of layers $n_{\rm layer}$ yields the total non-embedding parameter count.

Sources

equation

=12nlayerdmodel2 with the standard dattn=dff/4=dmodelformulae-sequenceabsent12subscript𝑛layersuperscriptsubscript𝑑model2 with the standard subscript𝑑attnsubscript𝑑ff4subscript𝑑model\displaystyle=12n_{\rm layer}d_{{\rm model}}^{2}\quad\text{ with the standard }\quad d_{\rm attn}=d_{\rm ff}/4=d_{{\rm model}} (2.1)

\displaystyle=12n_{\rm layer}d_{{\rm model}}^{2}\quad\text{ with the standard }\quad d_{\rm attn}=d_{\rm ff}/4=d_{{\rm model}}

Implementation detail

Illustrative Calculation

Let's calculate the parameter count for a model with:

$n_{\rm layer} = 12$
$d_{\rm model} = 768$

Using the formula: $N \approx 12 \times 12 \times 768^2 = 144 \times 589,824 = 84,934,656$

Sources

equation

=12nlayerdmodel2 with the standard dattn=dff/4=dmodelformulae-sequenceabsent12subscript𝑛layersuperscriptsubscript𝑑model2 with the standard subscript𝑑attnsubscript𝑑ff4subscript𝑑model\displaystyle=12n_{\rm layer}d_{{\rm model}}^{2}\quad\text{ with the standard }\quad d_{\rm attn}=d_{\rm ff}/4=d_{{\rm model}} (2.1)

\displaystyle=12n_{\rm layer}d_{{\rm model}}^{2}\quad\text{ with the standard }\quad d_{\rm attn}=d_{\rm ff}/4=d_{{\rm model}}

$n_{\rm layer}$: Number of layers in the Transformer model · scalar
$d_{\rm model}$: Hidden dimension size of the model · scalar
$d_{\rm attn}$: Attention dimension size · scalar
$d_{\rm ff}$: Feed-forward network intermediate dimension size · scalar
$N$: Total non-embedding parameter count · scalar