Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Hyperparameter and Shape Independence

Robustness to Architectural Hyperparameters

Sources

S3.F5

Figure 5: Performance depends very mildly on model shape when the total number of non-embedding parameters N𝑁N is held fixed. The loss varies only a few percent over a wide range of shapes. Small differences in parameter counts are compensated for by using the fit to L(N)𝐿𝑁L(N) as a baseline. Aspect ratio in particular can vary by a factor of 40 while only slightly impacting performance; an (nlayer,dmodel)=(6,4288)subscript𝑛layersubscript𝑑model64288(n_{\mathrm{layer}},d_{\mathrm{model}})=(6,4288) reaches a loss within 3% of the (48,1600)481600(48,1600) model used in [RWC+19].

Transformer performance depends very weakly on specific architectural hyperparameters—such as the number of layers $n_{\rm layer}$ , the number of attention heads $n_{\rm heads}$ , or the dimension of the feed-forward layers $d_{\rm ff}$ —when the total non-embedding parameter count $N$ is held constant.

Sources

S3.SS1.p1.10

Transformer performance depends very weakly on the shape parameters nlayer,nheadssubscript𝑛layersubscript𝑛headsn_{\rm layer},n_{\rm heads}, and dffsubscript𝑑ffd_{\rm ff} when we hold the total non-embedding parameter count N𝑁N fixed. To establish these results we trained models with fixed size while varying a single hyperparameter. This was simplest for the case of nheadssubscript𝑛headsn_{\rm heads}. When varying nlayersubscript𝑛layern_{\rm layer}, we simultaneously varied dmodelsubscript𝑑modeld_{{\rm model}} while keeping N≈12nlayerdmodel2𝑁12subscript𝑛layersuperscriptsubscript𝑑model2N\approx 12n_{\rm layer}d_{{\rm model}}^{2} fixed. Similarly, to vary dffsubscript𝑑ffd_{\rm ff} at fixed model size we also simultaneously varied the dmodelsubscript𝑑modeld_{{\rm model}} parameter, as required by the parameter counts in Table 1. Independence of nlayerssubscript𝑛layersn_{\rm layers} would follow if deeper Transformers effectively behave as ensembles of shallower models, as has been suggested for ResNets [VWB16]. The results are shown in Figure 5.

For example, the aspect ratio of the model can vary by a factor of 40 with only a negligible impact on performance. A very shallow and wide model can achieve performance within 3% of a much deeper, narrower baseline model of the same total size $N$ . This independence suggests that deeper Transformers may behave similarly to ensembles of shallower models.

Sources

S3.F5

Figure 5: Performance depends very mildly on model shape when the total number of non-embedding parameters N𝑁N is held fixed. The loss varies only a few percent over a wide range of shapes. Small differences in parameter counts are compensated for by using the fit to L(N)𝐿𝑁L(N) as a baseline. Aspect ratio in particular can vary by a factor of 40 while only slightly impacting performance; an (nlayer,dmodel)=(6,4288)subscript𝑛layersubscript𝑑model64288(n_{\mathrm{layer}},d_{\mathrm{model}})=(6,4288) reaches a loss within 3% of the (48,1600)481600(48,1600) model used in [RWC+19].

S3.SS1.p1.10

Transformer performance depends very weakly on the shape parameters nlayer,nheadssubscript𝑛layersubscript𝑛headsn_{\rm layer},n_{\rm heads}, and dffsubscript𝑑ffd_{\rm ff} when we hold the total non-embedding parameter count N𝑁N fixed. To establish these results we trained models with fixed size while varying a single hyperparameter. This was simplest for the case of nheadssubscript𝑛headsn_{\rm heads}. When varying nlayersubscript𝑛layern_{\rm layer}, we simultaneously varied dmodelsubscript𝑑modeld_{{\rm model}} while keeping N≈12nlayerdmodel2𝑁12subscript𝑛layersuperscriptsubscript𝑑model2N\approx 12n_{\rm layer}d_{{\rm model}}^{2} fixed. Similarly, to vary dffsubscript𝑑ffd_{\rm ff} at fixed model size we also simultaneously varied the dmodelsubscript𝑑modeld_{{\rm model}} parameter, as required by the parameter counts in Table 1. Independence of nlayerssubscript𝑛layersn_{\rm layers} would follow if deeper Transformers effectively behave as ensembles of shallower models, as has been suggested for ResNets [VWB16]. The results are shown in Figure 5.