One-sentence Summary

Researchers from FAIR at MSL Meta and New York University demonstrate that hyperparameter sensitivity, rather than model scale, confounds scaling laws, and introduce a model-centric methodology that recovers from small-scale experiments the large-scale result that pre-normalization works better as transformer models grow.

Key Contributions

• The paper shows that small models can exhibit scaling laws, but only after extensive hyperparameter tuning; it identifies hyperparameters as the main confounder and reports that well-tuned hyperparameters matter more than any other ingredient in the scaling law recipe.
• It explains the scale dependence of hyperparameter sensitivity by showing that sensitivity fades and the hyperparameter loss surface becomes lower dimensional as model size grows, which is why tuned small-scale configurations transfer to larger models with little adjustment.
• It develops a model-centric small-scale experiment methodology based on these insights and validates it by recovering the large-scale finding that pre-normalization works better as transformer models grow.

Introduction

Foundation models depend heavily on scale, which makes large-scale experiments extremely expensive, so researchers have long hoped to run cheap small-scale studies and transfer the findings to larger models. However, scaling laws have been unreliable at small scales: influential prior work reached contradictory conclusions about how parameters and data should scale, and later studies found that laws fit to models with roughly 100M parameters or fewer often fail to reproduce. The authors trace the core problem to hyperparameter sensitivity in small models and show that with rigorous tuning, scaling laws can extend down to about 4M parameters, where models train in under an hour on a single GPU. They explain this behavior through a hyperparameter loss surface analysis in which larger models become easier to tune. Their main contribution is a practical methodology for model-centric research: thoroughly explore hyperparameters at small scale, use diagnostics such as the noisy quadratic limit, scaling laws, and perplexity-capability correspondence to check assumptions, then transfer the resulting insights to large models with simple rules.

Method

The authors propose a methodology for estimating scaling laws from small-scale experiments, emphasizing that hyperparameter tuning is the critical ingredient. They first investigate the impact of parameter definitions, noting that choices regarding attention, embedding, and unembedding layers lead to minimal differences in the resulting scaling law. As shown in the figure below, various parameter counting philosophies yield reasonable scaling laws, though the authors prefer effective parameters (accounting for attention and unembedding but excluding embedding) to align directly with FLOPs per token, keeping $c = 6pd$c=6pd exact.

Beyond parameter definitions, the authors refine the estimation process through specific methodological choices. While a recognizable scaling law emerges from fitting the best run at each scale, precision improves significantly with further refinements. Tuning hyperparameters for each parameter-data budget reduces test MSE by 50%, and decaying the learning rate reduces it by 98%. Tying the scaling exponents $(\iota = \kappa)$(ι=κ), however, offers inconsistent results.

The core methodology for small-scale experiments rests on three principles: capabilities depend on pretraining loss alone, scaling laws extend to tiny scales with rigorous tuning, and models become less sensitive to hyperparameters as they scale. This implies that the hyperparameter loss surface becomes lower-dimensional and easier to optimize at larger scales. Consequently, small scales require extensive search, while large scales allow for easier adaptation.

To demonstrate this approach, the authors conduct a case study comparing pre-norm and post-norm transformer architectures. They sample configurations across well-separated scales (4M, 34M, and 134M parameters) to fit and validate scaling laws. They employ a sequence of qualitative diagnostics to understand the architectures. First, they check if the noisy quadratic limit emerges, indicating thorough tuning. The limit emerges readily for pre-norm but only deeper into the tail for post-norm, signaling the latter's hyperparameter sensitivity.

Second, they examine how hyperparameter sensitivity scales. As models grow, good configurations should fill more of the space. While both architectures show this trend, post-norm maintains a suboptimal peak, confirming persistent tuning difficulty.

Third, they verify that pretraining loss tracks downstream capabilities. Across various tasks and architectures, equal pretraining loss implies similar capabilities, validating the use of loss as a proxy for performance.

Finally, they estimate the scaling laws by tuning hyperparameters per budget and decaying the learning rate, fitting the laws jointly with tied irreducible error or leaving them free to differ to validate the extrapolation on the held-out scale.

Experiment

The experiments use random search across model scales from 4M to 268M parameters with warmup-stable-decay schedules to study scaling laws and hyperparameter loss surfaces. They find that methodological choices like parameter definitions matter little, while per-budget hyperparameter tuning and learning-rate decay greatly improve scaling-law precision, and that extensive tuning is essential at small scales. The hyperparameter loss surface becomes lower-dimensional and less sensitive as models grow, making larger models easier to tune. A case study comparing pre-norm and post-norm transformers shows pre-norm scales more reliably and is easier to tune, though extrapolating scaling laws remains statistically limited.