You are reading immutable version 1. The current guide may be newer.

Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resourcesā€”organized in teaching order.

All activities

Scaling Laws Assessment

Test your understanding

Check your understanding

1. Why does excluding the embedding parameters from the parameter count $N$ yield a cleaner power-law scaling relation for language model performance?
2. Write down the power-law equation for the loss $L(N)$ as a function of non-embedding parameters $N$, and state the approximate values of the exponent $\alpha_N$ and the constant $N_c$.
3. Under what condition does the language model loss scale as a clean power-law $L(D) \approx (D_c/D)^{\alpha_D}$ with dataset size $D$?
4. Write down the power-law equation for the loss $L(D)$ as a function of dataset size $D$ (in tokens), and state the approximate values of the exponent $\alpha_D$ and the constant $D_c$.
5. What are the approximate values of the exponent $\alpha_C^{\rm min}$ and the scale $C_{\mathrm{c}}^{\rm min}$ in the power-law scaling relation for the minimum training compute $C_{\rm min}$?
6. Write down the power-law equation for the loss $L(C_{\rm min})$ as a function of the minimum training compute $C_{\rm min}$, and explain what this relation represents.
7. How does varying architectural hyperparameters such as depth ($n_{\rm layer}$), number of attention heads ($n_{\rm heads}$), or feedforward dimension ($d_{\rm ff}$) affect performance when the total non-embedding parameter count $N$ is held fixed?
8. What hypothesis is mentioned in the paper to explain why Transformer performance is relatively independent of the number of layers ($n_{\rm layers}$) when the total parameter count is held fixed?
9. In the simultaneous scaling equation $L(N,D) = \left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}$, what does the equation simplify to in the limit of an infinitely large dataset ($D \to \infty$)?
10. Explain how the unified scaling law $L(N,D) = \left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}}$ predicts and models the phenomenon of overfitting when training a very large model on a relatively small dataset.
11. According to the empirical theory of batch size dependence, what is the primary advantage of training a model with a batch size $B \approx B_{\rm crit}$?
12. Write down the equation for the critical batch size $B_{\rm crit}(L)$ as a function of the loss $L$, and explain how $B_{\rm crit}$ changes as the model is trained to a lower loss.
13. When allocating an increased compute budget $C$ optimally, how do the optimal model size $N$ and the optimal number of training steps $S$ scale relative to $C$?
14. Based on the empirical relations $N(C_{\rm min}) \propto (C_{\rm min})^{0.73}$ and $S_{\rm min} \propto (C_{\rm min})^{0.03}$, explain how a developer should allocate a 10-fold increase in their training compute budget.