You are reading immutable version 3. The current guide may be newer.

Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resourcesā€”organized in teaching order.

All activities

Scaling Laws Assessment

Test your understanding

Check your understanding

1. Under what condition does language modeling performance exhibit a clean power-law relationship with each individual scale factor (model size $N$, dataset size $D$, and compute $C$)?
2. Explain what happens to the scaling of cross-entropy loss with respect to model size $N$ if the dataset size $D$ is kept extremely small and constant.
3. For a standard Transformer model with the standard configuration $d_{\rm attn} = d_{\rm ff}/4 = d_{\rm model}$, what is the approximate number of non-embedding parameters $N$?
4. Under what condition does the forward pass compute per token simplify to $C_{\rm forward} \approx 2N$ FLOPs, and why?
5. According to the empirical scaling laws, how does the minimum dataset size $D$ required to avoid overfitting scale with the non-embedding parameter count $N$?
6. Explain the practical implication of the sub-linear exponent ($0.74$) in the overfitting boundary $D \propto N^{0.74}$ for training very large models.
7. What is the relationship between the critical batch size $B_{\rm crit}$ and the loss $L$?
8. How does the critical batch size $B_{\rm crit}(L)$ behave as a model is trained to lower loss values, and what does this imply for batch size scheduling?
9. When allocating an increased compute budget $C_{\rm min}$ optimally, how do model size $N$ and training steps $S$ scale?
10. Why does the optimal compute allocation dictate that we should train very large models and stop them far before convergence, rather than training smaller models to full convergence?
11. At what approximate scale of parameters ($N^*$) and tokens ($D^*$) do the scaling law predictions for compute-efficient training and data-limited training begin to contradict each other?
12. What physical or informational interpretation do the authors conjecture for the intersection loss value $L^* \sim 1.7$ nats/token?