You are reading immutable version 2. The current guide may be newer.

Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Scaling Laws Comprehensive Assessment

Test your understanding

Check your understanding

1. Why are embedding parameters excluded from the definition of model size $N$ when analyzing scaling laws?
2. Given a Transformer model with $N$ non-embedding parameters, what is the estimated training compute $C$ in floating point operations (FLOPs) per training token, and what assumption is made regarding the context length $n_{\rm ctx}$ to justify this estimate?
3. Which of the following equations correctly represents the power-law relationship between cross-entropy loss $L(N)$ and non-embedding parameters $N$ when performance is not bottlenecked by other factors?
4. Write down the power-law equation for the cross-entropy loss $L(D)$ as a function of dataset size $D$ (in tokens) when not bottlenecked by model size or compute, and state the approximate value of the exponent $\alpha_D$.
5. Which of the following relations describes how the required dataset size $D$ scales with model size $N$ to avoid overfitting?
6. Explain why the common assumption that 'dataset size must scale linearly with model size to avoid overfitting' is incorrect, referencing the joint scaling relation between $D$ and $N$.
7. How does the critical batch size $B_{\rm crit}(L)$ scale with the cross-entropy loss $L$?
8. Write the mathematical definition of the minimum compute $C_{\rm min}(C)$ as a function of actual compute $C$, batch size $B$, and critical batch size $B_{\rm crit}(L)$, and state the condition under which this definition is primarily analyzed.
9. Under an optimal compute allocation strategy, how should the model size $N$ scale with the minimum compute budget $C_{\rm min}$?
10. Explain why training models to full convergence is not the most compute-efficient strategy when working with a fixed compute budget, according to the optimal scaling relations for $N$ and $S_{\rm min}$.
11. What is the conjectured physical meaning of the loss value $L^* \sim 1.7$ nats/token at the scaling intersection point?
12. Describe the apparent contradiction that arises at very large scales between compute-efficient training and natural language data limits.