Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

All activities

Scaling Laws Assessment

Test your understanding

Check your understanding

1. When not bottlenecked by other factors, what mathematical relationship describes how language modeling performance (cross-entropy loss) scales with model size $N$, dataset size $D$, and compute $C$?
2. Under what condition do the individual power-law scaling trends for model size $N$, dataset size $D$, and compute $C$ hold true?
3. Why do the authors exclude vocabulary and positional embedding parameters from the model parameter count $N$ when analyzing scaling laws?
4. Which of the following is a key reason for excluding embedding parameters from the parameter count $N$?
5. How does varying the aspect ratio (depth vs. width) of a Transformer affect its performance if the total number of non-embedding parameters $N$ is held constant?
6. According to the paper's findings, which of the following statements is true regarding architectural hyperparameters (like $n_{\rm layer}$, $n_{\rm heads}$, and $d_{\rm ff}$) when the total non-embedding parameter count $N$ is held fixed?
7. Based on the joint scaling law $L(N,D)$, what specific ratio of model size $N$ to dataset size $D$ governs the extent of overfitting?
8. What happens to the test loss $L(N,D)$ of a model trained on a small, fixed dataset size $D$ as the model size $N$ is increased indefinitely?
9. How does the critical batch size $B_{\rm crit}$ scale as performance improves (i.e., as loss decreases)? Describe the empirical rule of thumb identified in the paper.
10. Does the critical batch size $B_{\rm crit}$ depend directly on the model size $N$?
11. When scaling up the compute budget by 10x, by what factors should the optimal model size $N$ and the number of data examples processed grow, respectively?
12. How does the optimal batch-adjusted number of optimization steps change as the compute budget increases?