Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Optimal Allocation of Compute Budget

Figure 5: Hyperparameter Independence

Performance depends very mildly on model shape when the total number of non-embedding parameters $N$ is held fixed. The loss varies only a few percent over a wide range of shapes. Aspect ratio in particular can vary by a factor of 40 while only slightly impacting performance; an $(n_{\mathrm{layer}}, d_{\mathrm{model}}) = (6, 4288)$ model reaches a loss within 3% of the $(48, 1600)$ model.

Sources

S3.F5

Figure 5: Performance depends very mildly on model shape when the total number of non-embedding parameters N𝑁N is held fixed. The loss varies only a few percent over a wide range of shapes. Small differences in parameter counts are compensated for by using the fit to L(N)𝐿𝑁L(N) as a baseline. Aspect ratio in particular can vary by a factor of 40 while only slightly impacting performance; an (nlayer,dmodel)=(6,4288)subscript𝑛layersubscript𝑑model64288(n_{\mathrm{layer}},d_{\mathrm{model}})=(6,4288) reaches a loss within 3% of the (48,1600)481600(48,1600) model used in [RWC+19].

Reported values
Measure	Value

Sources

S3.F5

Figure 5: Performance depends very mildly on model shape when the total number of non-embedding parameters N𝑁N is held fixed. The loss varies only a few percent over a wide range of shapes. Small differences in parameter counts are compensated for by using the fit to L(N)𝐿𝑁L(N) as a baseline. Aspect ratio in particular can vary by a factor of 40 while only slightly impacting performance; an (nlayer,dmodel)=(6,4288)subscript𝑛layersubscript𝑑model64288(n_{\mathrm{layer}},d_{\mathrm{model}})=(6,4288) reaches a loss within 3% of the (48,1600)481600(48,1600) model used in [RWC+19].