Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Transformer Parameter and Compute Scaling

Parameter and Compute Scaling of Transformers

Sources

section

2.1 Parameter and Compute Scaling of Transformers

When analyzing the scaling behavior of Transformer models, we define the model size $N$ as the number of non-embedding parameters. This means we explicitly exclude the parameters from the vocabulary embedding matrix ( $n_{\rm vocab} d_{\rm model}$ ) and the positional embeddings ( $n_{\rm ctx} d_{\rm model}$ ), as well as biases and other sub-leading terms. Excluding these embedding parameters yields significantly cleaner scaling laws because embedding parameters are highly dependent on vocabulary size and context length, which do not scale in the same manner as the core representation capacity of the network.

Sources

S2.SS1.p2.1

We use N𝑁N to denote the model size, which we define as the number of non-embedding parameters

S2.SS1.p2.4

where we have excluded biases and other sub-leading terms. Our models also have nvocabdmodelsubscript𝑛vocabsubscript𝑑modeln_{\rm vocab}d_{{\rm model}} parameters in an embedding matrix, and use nctxdmodelsubscript𝑛ctxsubscript𝑑modeln_{\rm ctx}d_{{\rm model}} parameters for positional embeddings, but we do not include these when discussing the ‘model size’ N𝑁N; we will see that this produces significantly cleaner scaling laws.

For standard Transformer architectures where the model depth and width are large relative to the context length (specifically $d_{\rm model} \gg n_{\rm ctx}/12$ ), the context-dependent computational cost per token is negligible. Accounting for both the forward pass and the backward pass (which takes approximately twice the compute of the forward pass), the total estimated non-embedding training compute $C$ is approximated by:

Sources

S2.SS1.p4.3

For contexts and models with dmodel>nctx/12subscript𝑑modelsubscript𝑛ctx12d_{{\rm model}}>n_{\rm ctx}/12, the context-dependent computational cost per token is a relatively small fraction of the total compute. Since we primarily study models where dmodel≫nctx/12much-greater-thansubscript𝑑modelsubscript𝑛ctx12d_{\rm model}\gg n_{\rm ctx}/12, we do not include context-dependent terms in our training compute estimate. Accounting for the backwards pass (approximately twice the compute as the forwards pass), we then define the estimated non-embedding compute as C≈6N𝐶6𝑁C\approx 6N floating point operators per training token.

$C \approx 6N$

Sources

S2.SS1.p4.3

For contexts and models with dmodel>nctx/12subscript𝑑modelsubscript𝑛ctx12d_{{\rm model}}>n_{\rm ctx}/12, the context-dependent computational cost per token is a relatively small fraction of the total compute. Since we primarily study models where dmodel≫nctx/12much-greater-thansubscript𝑑modelsubscript𝑛ctx12d_{\rm model}\gg n_{\rm ctx}/12, we do not include context-dependent terms in our training compute estimate. Accounting for the backwards pass (approximately twice the compute as the forwards pass), we then define the estimated non-embedding compute as C≈6N𝐶6𝑁C\approx 6N floating point operators per training token.

Deep dive

This simple relation ( $C \approx 6N$ floating point operations per training token) provides a direct, clean bridge between parameter count and computational budget, allowing us to analyze scaling trends without the noise of context-dependent overhead.

Sources

S2.SS1.p4.3

For contexts and models with dmodel>nctx/12subscript𝑑modelsubscript𝑛ctx12d_{{\rm model}}>n_{\rm ctx}/12, the context-dependent computational cost per token is a relatively small fraction of the total compute. Since we primarily study models where dmodel≫nctx/12much-greater-thansubscript𝑑modelsubscript𝑛ctx12d_{\rm model}\gg n_{\rm ctx}/12, we do not include context-dependent terms in our training compute estimate. Accounting for the backwards pass (approximately twice the compute as the forwards pass), we then define the estimated non-embedding compute as C≈6N𝐶6𝑁C\approx 6N floating point operators per training token.