Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

12 activities

Lesson

At a glance

Overview of Scaling Laws for Language Models

Scaling Laws for Neural Language Models

Lesson

Transformer Parameter and Compute Scaling

Parameter and Compute Scaling of Transformers

Lesson

Power-law scaling of loss with model size

Basic Power Laws of Scaling

This equation models the power-law scaling relationship between the cross-entropy loss L(N) and the number of non-embedding parameters N of a neural language model, assuming performance is not bottlenecked by dataset size or compute limits.

Lesson

Power-law scaling of loss with dataset size

Basic Power Laws of Scaling

This equation models the power-law scaling relationship between the cross-entropy loss L(D) and the dataset size D (measured in tokens) when training a language model that is not bottlenecked by model capacity or compute budget.

Lesson

Power-law scaling of loss with minimum compute

Basic Power Laws of Scaling

This equation models the power-law scaling of the cross-entropy loss L as a function of the minimum compute C rm min required to reach that loss, assuming training is not bottlenecked by dataset size or model capacity.

Lesson

Critical batch size as a power law of loss

Basic Power Laws of Scaling

The critical batch size B rm crit (L) determines the boundary where increasing the batch size further yields diminishing returns in terms of optimization speedup per step. This relationship is modeled as a power law of the cross-entropy loss L .

Lesson

Joint scaling law of loss with model and dataset size

Overfitting and Joint Scaling L(N, D)

The joint scaling law L(N, D) models the cross-entropy loss of a language model as a function of both the number of non-embedding parameters N and the dataset size D (measured in tokens). This formulation captures how performance scales when both resources are finite, describing the transition into the overfitting regime.

Lesson

Critical Batch Size and Training Dynamics

Critical Batch Size Scaling

Lesson

Optimal model size scaling with compute

Optimal Allocation of Compute Budgets

The optimal model size N scales with the minimum compute budget C rm min as a power-law, where the exponent is approximately 0.73 .

Lesson

Intersection point of scaling laws

Optimal Allocation of Compute Budgets

The intersection point where the scaling laws suggest a potential contradiction or transition is characterized by a compute budget C^ sim 10^4 PF-Days, model size N^ sim 10^ 12 parameters, dataset size D^ sim 10^ 12 tokens, and loss L^ sim 1.7 nats/token.

Lesson

Scaling Contradictions and the Entropy Conjecture

Contradictions and Conjectures at Scale

Quiz

Test your understanding

Scaling Laws Comprehensive Assessment

12 questions grounded in this paper section.