Study the paper
Scaling Laws for Neural Language Models
Lessons, visuals, quizzes, flashcards, and resourcesāorganized in teaching order.
11 activities
At a glance
Overview of Scaling Laws for Language ModelsScaling Laws for Neural Language Models
LessonModel Shape and Hyperparameter Independence
Model Shape and Hyperparameter IndependenceIndependence of Transformer Shape Parameters
LessonPower-law scaling with non-embedding parameters
The Infinite Data Limit and OverfittingThis equation models the empirical power-law scaling of the cross-entropy loss L(N) of a language model as a function of its number of non-embedding parameters N . By excluding embedding parameters, the scaling law exhibits a cleaner power-law behavior over several orders of magnitude.
LessonPower-law scaling with dataset size
The Infinite Data Limit and OverfittingThis equation models the power-law scaling of language model performance (measured by cross-entropy loss) as a function of the dataset size D (in tokens), assuming that the model capacity and training compute are not bottlenecking performance.
LessonPower-law scaling with minimum compute
The Infinite Data Limit and OverfittingThis equation describes the power-law scaling of the cross-entropy loss L as a function of the minimum training compute C rm min (measured in PF-days) required to reach a given performance level.
LessonUnified scaling law for parameters and dataset size
The Infinite Data Limit and OverfittingThe unified scaling law models the cross-entropy loss L(N,D) as a joint function of the number of non-embedding parameters N and the dataset size D (measured in tokens). This formulation captures how performance scales when both resources are finite, predicting the onset of overfitting when one resource is bottlenecked relative to the other.
LessonOptimal compute allocation scaling
The Infinite Data Limit and OverfittingThis equation describes the optimal scaling of model parameters N , batch size B , and training steps S as power-law functions of the total compute budget C . It also defines the total dataset size D in tokens as the product of batch size and training steps.
LessonCritical batch size power law
Critical Batch Size and Training DynamicsThe critical batch size B rm crit (L) determines the optimal batch size for training language models. It represents the point of diminishing returns where increasing the batch size further yields minimal parallelization speedup per unit of compute. This relationship is modeled as a power law of the cross-entropy loss L .
LessonFigure 5: Shape parameter independence
Optimal Allocation of Compute BudgetsPerformance depends very mildly on model shape when the total number of non-embedding parameters N is held fixed. The loss varies only a few percent over a wide range of shapes. Aspect ratio in particular can vary by a factor of 40 while only slightly impacting performance; an (n mathrm layer ,d mathrm model )=(6,4288) model reaches a loss within 3% of the (48,1600) model.
LessonFigure 14: Optimal model size and steps vs compute
Optimal Allocation of Compute BudgetsEach value of the compute budget C rm min has an associated optimal model size N . Optimal model size grows very rapidly with C rm min , increasing by 5x for each 10x increase in compute. The number of data examples processed makes up the remainder of the increase, growing relatively modestly by only 2x.
QuizTest your understanding
Scaling Laws Assessment14 questions grounded in this paper section.