Study the paper

Scaling Laws for Neural Language Models

Lessons, visuals, quizzes, flashcards, and resources—organized in teaching order.

Model Shape and Hyperparameter Independence

Independence of Transformer Shape Parameters

Sources

section

3.1 Approximate Transformer Shape and Hyperparameter Independence

An important finding in scaling analysis is that Transformer performance depends very weakly on individual shape hyperparameters—such as the number of layers ( $n_{\rm layer}$ ), attention heads ( $n_{\rm heads}$ ), or the feed-forward dimension ( $d_{\rm ff}$ )—provided that the total number of non-embedding parameters $N$ is held constant.

Sources

S3.SS1.p1.10

Transformer performance depends very weakly on the shape parameters nlayer,nheadssubscript𝑛layersubscript𝑛headsn_{\rm layer},n_{\rm heads}, and dffsubscript𝑑ffd_{\rm ff} when we hold the total non-embedding parameter count N𝑁N fixed. To establish these results we trained models with fixed size while varying a single hyperparameter. This was simplest for the case of nheadssubscript𝑛headsn_{\rm heads}. When varying nlayersubscript𝑛layern_{\rm layer}, we simultaneously varied dmodelsubscript𝑑modeld_{{\rm model}} while keeping N≈12nlayerdmodel2𝑁12subscript𝑛layersuperscriptsubscript𝑑model2N\approx 12n_{\rm layer}d_{{\rm model}}^{2} fixed. Similarly, to vary dffsubscript𝑑ffd_{\rm ff} at fixed model size we also simultaneously varied the dmodelsubscript𝑑modeld_{{\rm model}} parameter, as required by the parameter counts in Table 1. Independence of nlayerssubscript𝑛layersn_{\rm layers} would follow if deeper Transformers effectively behave as ensembles of shallower models, as has been suggested for ResNets [VWB16]. The results are shown in Figure 5.

To demonstrate this, models were trained by varying a single hyperparameter while holding the overall scale fixed. For example, when varying $n_{\rm layer}$ , the model dimension $d_{\rm model}$ was simultaneously adjusted to keep the parameter count $N \approx 12 n_{\rm layer} d_{\rm model}^2$ constant. Across a wide range of aspect ratios (varying by up to a factor of 40), the cross-entropy loss varies by only a few percent.

Sources

S3.SS1.p1.10

Transformer performance depends very weakly on the shape parameters nlayer,nheadssubscript𝑛layersubscript𝑛headsn_{\rm layer},n_{\rm heads}, and dffsubscript𝑑ffd_{\rm ff} when we hold the total non-embedding parameter count N𝑁N fixed. To establish these results we trained models with fixed size while varying a single hyperparameter. This was simplest for the case of nheadssubscript𝑛headsn_{\rm heads}. When varying nlayersubscript𝑛layern_{\rm layer}, we simultaneously varied dmodelsubscript𝑑modeld_{{\rm model}} while keeping N≈12nlayerdmodel2𝑁12subscript𝑛layersuperscriptsubscript𝑑model2N\approx 12n_{\rm layer}d_{{\rm model}}^{2} fixed. Similarly, to vary dffsubscript𝑑ffd_{\rm ff} at fixed model size we also simultaneously varied the dmodelsubscript𝑑modeld_{{\rm model}} parameter, as required by the parameter counts in Table 1. Independence of nlayerssubscript𝑛layersn_{\rm layers} would follow if deeper Transformers effectively behave as ensembles of shallower models, as has been suggested for ResNets [VWB16]. The results are shown in Figure 5.

S3.F5

Figure 5: Performance depends very mildly on model shape when the total number of non-embedding parameters N𝑁N is held fixed. The loss varies only a few percent over a wide range of shapes. Small differences in parameter counts are compensated for by using the fit to L(N)𝐿𝑁L(N) as a baseline. Aspect ratio in particular can vary by a factor of 40 while only slightly impacting performance; an (nlayer,dmodel)=(6,4288)subscript𝑛layersubscript𝑑model64288(n_{\mathrm{layer}},d_{\mathrm{model}})=(6,4288) reaches a loss within 3% of the (48,1600)481600(48,1600) model used in [RWC+19].

Key Takeaway

Because performance is highly insensitive to architectural hyperparameters, researchers can choose Transformer shapes based on hardware efficiency and parallelization constraints rather than tuning for optimal loss.

Sources

S3.SS1.p1.10

Transformer performance depends very weakly on the shape parameters nlayer,nheadssubscript𝑛layersubscript𝑛headsn_{\rm layer},n_{\rm heads}, and dffsubscript𝑑ffd_{\rm ff} when we hold the total non-embedding parameter count N𝑁N fixed. To establish these results we trained models with fixed size while varying a single hyperparameter. This was simplest for the case of nheadssubscript𝑛headsn_{\rm heads}. When varying nlayersubscript𝑛layern_{\rm layer}, we simultaneously varied dmodelsubscript𝑑modeld_{{\rm model}} while keeping N≈12nlayerdmodel2𝑁12subscript𝑛layersuperscriptsubscript𝑑model2N\approx 12n_{\rm layer}d_{{\rm model}}^{2} fixed. Similarly, to vary dffsubscript𝑑ffd_{\rm ff} at fixed model size we also simultaneously varied the dmodelsubscript𝑑modeld_{{\rm model}} parameter, as required by the parameter counts in Table 1. Independence of nlayerssubscript𝑛layersn_{\rm layers} would follow if deeper Transformers effectively behave as ensembles of shallower models, as has been suggested for ResNets [VWB16]. The results are shown in Figure 5.

S3.F5

Figure 5: Performance depends very mildly on model shape when the total number of non-embedding parameters N𝑁N is held fixed. The loss varies only a few percent over a wide range of shapes. Small differences in parameter counts are compensated for by using the fit to L(N)𝐿𝑁L(N) as a baseline. Aspect ratio in particular can vary by a factor of 40 while only slightly impacting performance; an (nlayer,dmodel)=(6,4288)subscript𝑛layersubscript𝑑model64288(n_{\mathrm{layer}},d_{\mathrm{model}})=(6,4288) reaches a loss within 3% of the (48,1600)481600(48,1600) model used in [RWC+19].