Scaling Laws for Neural Language Models

Overview of Scaling Laws in Language Models

At a glance

Scaling Laws in Language Models

Transformer Parameter Count Approximation

This equation estimates the total number of non-embedding parameters in a standard Transformer model. It assumes standard scaling relationships between the model dimension, attention dimension, and feed-forward dimension.

Transformer Forward Pass FLOPs Estimate

This equation estimates the computational cost (in FLOPs per token) of a forward pass through a standard Transformer model. It accounts for both the parameter-dependent operations (such as feed-forward networks and projections) and the context-dependent attention operations.

Unified Loss Equation L(N,D)

The unified loss equation L(N,D) models the joint dependence of the cross-entropy loss on both the number of non-embedding parameters N and the dataset size D (measured in tokens). It captures how overfitting occurs when either resource is constrained relative to the other.

Optimal Compute Allocation Scaling Relations

This set of scaling relations determines how to optimally allocate an increased compute budget C among the model size (number of non-embedding parameters N ), the batch size B , and the number of training steps S . The total number of tokens processed during training is given by D = B cdot S .

Transformer Parameter Count Approximation: step by step

Follow the existing bounded walkthrough in its intended sequence.

Transformer Forward Pass FLOPs Estimate: step by step

Follow the existing bounded walkthrough in its intended sequence.

Optimal Compute Allocation Scaling Relations: step by step

Follow the existing bounded walkthrough in its intended sequence.

Unified Loss Equation L(N,D): step by step

Follow the existing bounded walkthrough in its intended sequence.

The Infinite Data Limit and Overfitting

Joint Scaling and Overfitting

Critical Batch Size and Training Dynamics

Critical Batch Size Power Law

The critical batch size B rm crit (L) determines the optimal batch size for training neural language models. It represents the threshold where further increasing the batch size yields diminishing returns in terms of training speedup per step. This relationship is modeled as a power law of the cross-entropy loss L .

Critical Batch Size and Training Dynamics

Critical Batch Size Power Law: step by step

Follow the existing bounded walkthrough in its intended sequence.

Optimal Allocation of Compute Budgets

Language modeling performance improves smoothly as we increase the model size, dataset size, and amount of compute used for training. Empirical performance has a power-law relationship with each individual factor when not bottlenecked by the other two.