Study the paper
Scaling Laws for Neural Language Models
Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.
14 activities
At a glance
Overview of Scaling Laws for Language ModelsScaling Laws for Language Models
LessonOverview of Scaling Laws for Language Models
Overview of Scaling Laws for Language ModelsPower-Law Scaling of Language Models
LessonHyperparameter and Shape Independence
Hyperparameter and Shape IndependenceRobustness to Architectural Hyperparameters
LessonJoint Scaling Law L(N,D)
Overfitting and the Infinite Data LimitThe joint scaling law L(N,D) models the simultaneous dependence of the cross-entropy loss on both the model parameter count N and the dataset size D . It captures how performance scales when both resources are finite, incorporating the effects of overfitting.
LessonOptimal Allocation Scaling
Overfitting and the Infinite Data LimitThis equation describes the optimal scaling of model parameters ( N ), batch size ( B ), and training steps ( S ) as power-law functions of the total compute budget ( C ). It also defines the total dataset size in tokens ( D ) as the product of batch size and training steps.
VisualJoint Scaling Law L(N,D): step by step
Overfitting and the Infinite Data LimitFollow the existing bounded walkthrough in its intended sequence.
VisualOptimal Allocation Scaling: step by step
Overfitting and the Infinite Data LimitFollow the existing bounded walkthrough in its intended sequence.
LessonCritical Batch Size Power Law
Critical Batch Size and Training DynamicsThe critical batch size B rm crit (L) determines the optimal batch size for training neural language models. It represents the threshold where increasing the batch size further yields diminishing returns in terms of training speedup per unit of compute. This relationship is modeled as a power law of the cross-entropy loss L .
VisualCritical Batch Size Power Law: step by step
Critical Batch Size and Training DynamicsFollow the existing bounded walkthrough in its intended sequence.
LessonFigure 5: Hyperparameter Independence
Optimal Allocation of Compute BudgetPerformance depends very mildly on model shape when the total number of non-embedding parameters N is held fixed. The loss varies only a few percent over a wide range of shapes. Aspect ratio in particular can vary by a factor of 40 while only slightly impacting performance; an (n mathrm layer , d mathrm model ) = (6, 4288) model reaches a loss within 3% of the (48, 1600) model.
LessonFigure 9: Overfitting and Dataset Size
Optimal Allocation of Compute BudgetThe early-stopped test loss L(N,D) depends predictably on the dataset size D and model size N . For large D , performance is a straight power law in N . For a smaller fixed D , performance stops improving as N increases and the model begins to overfit. The extent of overfitting depends predominantly on the ratio N^ alpha N / alpha D / D .
LessonFigure 14: Optimal Allocation of Compute
Optimal Allocation of Compute BudgetEach value of the compute budget C rm min has an associated optimal model size N . Optimal model size grows very rapidly with C rm min , increasing by 5x for each 10x increase in compute. The number of data examples processed makes up the remainder of the increase, growing relatively modestly by only 2x.
QuizTest your understanding
Scaling Laws Assessment12 questions grounded in this paper section.
ResourceFurther learning
Scaling Laws Assessment3 supplementary resources for this paper section.