Q Labs Outlines Training Rules for Quadrillion-Parameter Transformers
Q Labs Research has proposed a path toward quadrillion-parameter transformer training to address the sample-efficiency gap between human and artificial learning. In a 2026 paper by Ethan Baron and colleagues, the researchers argue that today’s large language models are far too small relative to human-scale parameter counts. They compare Chinchilla-style scaling, which suggests about 1.5 trillion parameters for 30 trillion tokens, with the human brain’s estimated 100 trillion synaptic parameters trained on roughly 200 million tokens. This creates a tokens-per-parameter gap of several orders of magnitude. The paper focuses on the data-constrained regime, where compute is growing faster than available high-quality text and models must train over multiple epochs. Instead of holding model size fixed, the team finds that the compute-optimal model size continues to increase as the dataset is reused. They fit a scaling law that predicts loss from unique tokens, model size, epochs, and ensemble members. A key component is hyperparameter transfer: the law prescribes how learning rate and, especially, weight decay should increase with model size and epoch count. Previous rules do not account for epochs and can miss optimal settings. The researchers also treat ensembling as a first-class scaling axis. Snapshot ensembling averages checkpoints from a single training run, using cyclic learning rates to produce diverse strong solutions without extra training compute. Deep ensembling, which averages independently trained models, becomes useful when compute is large relative to data. In the data-constrained setting, adding ensemble members reduces overfitting instead of worsening it. Extrapolated to 5 trillion tokens at 10^29 FLOPs, the prescription calls for an ensemble with about 300 parameters per token, nearly 7,000 times Chinchilla’s ratio. The authors do not claim such models are already trainable, but say the result points toward much larger networks and new algorithms, hardware, and training recipes needed to improve sample efficiency.
