One-sentence Summary

IQuest Research introduces the Loopie series of Mixture-of-Experts models, comprising a 20B-parameter model with 2B active parameters and a 6B-parameter model with 0.6B active parameters, which overcome the historical challenge that scaling parameter count outperforms looping by substantially outperforming vanilla Transformer baselines under the same compute budget, and, through a novel post-training method, achieve frontier-level reasoning performance.

Key Contributions

• Loopie, a family of looped Mixture-of-Experts language models, combines layer-loop recurrence with a hardware-aware scaling recipe to make recurrent depth competitive under matched pre-training compute budgets.
• Extensive ablation studies demonstrate that Loopie models substantially outperform compute-matched vanilla Transformer baselines across model scales.
• A post-training pipeline of supervised pre-training and reinforcement learning equips Loopie with strong mathematical reasoning and coding abilities, achieving frontier-level reasoning performance.

Introduction

Looped Transformers reuse the same layers across recurrent steps, enabling strong performance on reasoning tasks with far fewer parameters than vanilla depth-scaled models. However, prior work compares looped and non-looped models at equal parameter counts rather than under matched pre-training compute budgets, ignoring that looping multiplies training FLOPs. Moreover, most studies focus on dense architectures, not the Mixture-of-Experts (MoE) designs that dominate frontier models. The authors address these gaps by introducing Loopie, a family of looped MoE language models, and a compute-matched scaling recipe that jointly optimizes width, depth, and recurrence to stay competitive under a fixed training compute budget. A novel post-training pipeline further equips these models with strong reasoning abilities.

Dataset

For Stage 2 annealing, the authors construct a high-quality mixture from several Nemotron pre-training datasets, totaling approximately 1.26 trillion tokens. The mixture combines SFT-style data, synthetic reasoning, code, synthetic web data, and math data in the following proportions:

• Nemotron-pre-training-SFT-v1 (351B tokens, 27.8%): A diverse SFT-style dataset of synthetic and curated examples covering STEM, academic, code, mathematics, and reasoning domains with multilingual coverage. STEM data is expanded from high-quality math and science seeds via iterative generation with Qwen3 and DeepSeek models, producing harder and more varied questions with solutions. It also contains academic QA pairs synthesized from undergraduate/graduate texts and MMLU-style general QA and reasoning data.
• Nemotron-pre-training-Specialized-v1 (277B tokens, 21.9%): Synthetic data for STEM reasoning, scientific coding, cross-domain coding, synthetic Wikipedia, and synthetic mathematics textbooks. Scientific coding and cross-domain subsets introduce graduate- or research-level programming tasks with structured solutions, aiming to strengthen scientific reasoning, mathematical abstraction, and code generation.
• Nemotron-pre-training-Code-v2 (262B tokens, 20.7%): A 60% random sample of the full Code-v2 dataset. Contains recent GitHub source code, synthetic code-grounded QA, student-teacher dialogues, code-review dialogues, and LLM-rewritten or transpiled source code to increase stylistic diversity and expose the model to semantically equivalent code variants.
• High-quality synthetic web data (197B tokens + 25B tokens, 15.6% + 2.0%): A 16% sample of Nemotron-CC-v2-High-Quality-Synthetic (197B tokens) derived from English web-crawl documents rephrased with Qwen3-30B-A3B, plus 25B tokens from Nemotron-CC-v2.1-High-Quality-Synthetic that extends the synthetic corpus with newer Common Crawl snapshots and rephrased medium- to high-quality documents.
• High-quality web data (25B tokens, 2.0%): From Nemotron-CC-v2.1-High-Quality, incorporating recent Common Crawl snapshots and high-quality translated data, with LLM-based filtering to remove uninformative translated documents. Preserves natural web text exposure during annealing.
• Mathematics data (126B tokens, 10.0%): From Nemotron-CC-Math-v1, retaining only documents with quality scores ≥ 4. Built from Common Crawl using a pipeline that preserves equations, converts notation to standardized LaTeX, and removes noise, strengthening mathematical reasoning and symbolic problem-solving.

How the data is used: The mixture is the sole training corpus for the Stage 2 annealing phase. All subsets are combined in the listed token ratios with no explicit mention of a separate validation split. Pre-processing includes random sampling (for code and synthetic web subsets), quality-score filtering (for math), and synthetic augmentation (rephrasing, rewriting, and LaTeX normalization), all of which are applied before the annealing run.

Method

The authors leverage a decoder-only Mixture-of-Experts (MoE) Transformer backbone, largely following the Qwen3-MoE architecture. The key architectural innovation lies in the application of recurrent computation. Instead of the traditional model-loop pattern used in prior looped language models, where the entire Transformer stack is unrolled recurrently, Loopie adopts a layer-loop pattern.

As shown in the figure below:

In the layer-loop pattern, each layer is applied recurrently before the computation moves to the next layer. For a model with $N$N layers and $R$R loop steps, the computation order applies Layer 1 $R$R times, then Layer 2 $R$R times, and so on. This contrasts with model-loop, which traverses all layers and then repeats the entire stack. The authors demonstrate that layer-loop offers three distinct advantages for large-scale pre-training. First, it achieves better empirical scaling. While layer-loop may initially trail model-loop on downstream benchmarks, it overtakes it as the training budget increases.

Refer to the performance comparison diagram:

Second, layer-loop provides infrastructure-friendly execution. Repeated applications of the same layer are adjacent in the computation graph, shortening reuse distance for parameters and simplifying activation checkpointing and pipeline parallelism. Third, it creates a natural parameter-sharing pattern. Layer-loop reuses a layer at adjacent effective depths, resulting in a more coherent parameter-sharing pattern compared to model-loop, which asks a single parameter set to accommodate hidden states at widely separated effective depths.

To justify recurrence as a scaling strategy, the model must compete with vanilla Transformers trained under the same pre-training compute budget. The authors formalize the Loopie Recipe to address this. Starting from a strong non-recurrent MoE reference, the recipe involves three steps: constructing a recurrent seed model by halving the number of stored layers, executing each stored layer twice using layer-loop ($R=2$R=2), and using the resulting memory headroom to double the per-device microbatch size. The measured training efficiency gain is then reinvested into additional model capacity while keeping the optimizer-step time matched to the reference model.

The authors choose $R=2$R=2 because, under a fixed compute budget, the marginal return of additional recurrence decays rapidly. Increasing loop steps multiplies the per-token training cost. To isolate the benefit of recurrence, they compare models using $N\times$N× layer-loop steps against those with $N\times$N× as many stored layers.

As illustrated in the scaling comparison figure:

The results indicate that the marginal benefit of recurrence is largest at $R=2$R=2. This setting allows each layer to perform one ordinary transformation followed by one local refinement step, keeping the compute multiplier small enough for scalable pre-training while preserving training throughput.

The pre-training process consists of two stages. In Stage 1, the models undergo large-scale pre-training from scratch on high-quality data for multiple epochs. In Stage 2, the authors conduct high-quality data annealing using a mixture of synthetic, STEM, code, mathematical reasoning, and web data.

The composition of this annealing data pool is visualized below:

This mixture combines high-quality SFT-style data, synthetic reasoning data, code data, and math data to strengthen scientific reasoning, mathematical abstraction, and code generation capabilities during the annealing phase.

The post-training recipe transforms the base model into the final Loopie Thinking model through two main stages. First, the authors introduce Supervised Pre-Training (SPT), a regime that combines the supervision pattern of supervised fine-tuning (SFT) with the optimization scale of language-model pre-training. SPT uses the same token-level objective as SFT, where only supervised target tokens contribute to the loss, but applies it at a pre-training scale with large global batch sizes and token budgets. This approach mitigates catastrophic forgetting and improves performance on both general knowledge and reasoning metrics.

The trends in pre-training metrics during SPT are shown in the following figure:

Following SPT, the authors apply a reinforcement learning stage using Group Sequence Policy Optimization (GSPO). This stage enhances reasoning capabilities and aligns the model to produce reliable thinking traces. The training is conducted sequentially on mathematical tasks and then coding tasks, utilizing sequence-level clipped objectives and prompt-level dynamic filtering to focus optimization on prompts that yield non-degenerate policy gradients.

Experiment

Loopie recurrent MoE models were evaluated against compute-matched vanilla baselines, initially lagging but later overtaking and maintaining this advantage across a scaling ladder from 0.15B to 1B parameters. Ablations confirm that the specific layer-loop recurrence schedule is critical, and that two loop steps offer the best trade-off between iterative refinement and compute efficiency. Post-training evaluations further demonstrate that Loopie achieves substantially higher token efficiency, with a 20B model trained on 3.5T tokens matching or exceeding models trained on 25T tokens while also delivering strong reasoning performance.

Starting from a Qwen3-like 30B-A3B reference, halving the layer count and applying layer-loop with R=2 preserves the compute proxy while halving activation memory. The memory headroom allows doubling the per-device microbatch size, and the resulting throughput gain is reinvested into wider and deeper models, yielding up to 1.65× the block work proxy with only 69% of the reference activation memory, all while keeping optimizer-step time matched to the reference. A seed model with half the layers and R=2 loop achieves the same compute proxy as the reference but uses only 50% of its activation memory. Across the candidate configurations, the compute proxy increases from 1.00× to 1.65× while activation memory remains well below the reference, peaking at 0.69×, demonstrating that memory savings from recurrence can be efficiently traded for additional model capacity.

Supervised pre-training (SPT) applies the loss only to supervised target tokens, like supervised fine-tuning (SFT), but uses the large batch sizes, long sequences, and token budgets typical of language-model pre-training (PT). This regime simultaneously improves pre-training and reasoning metrics, avoids the overfitting observed in multi-epoch SFT, and decouples the choice of loss-bearing tokens from the optimization scale. SPT improves both pre-training and reasoning metrics, while SFT degrades pre-training metrics and PT leaves reasoning largely unchanged. SPT uses global batch sizes of at least 1,024 and sequences of at least 128K tokens, matching the optimization scale of pre-training and preventing overfitting across multiple epochs.

Loopie-20B-A2B matches or surpasses similarly sized MoE reasoning models on knowledge and general benchmarks while using less than one-seventh of their pretraining tokens. It also achieves strong post-training reasoning results, ranking second on AIME 2024 and AMC, and tying a larger-scale model on AIME 2025. The model demonstrates substantially higher token efficiency under the same pretraining data. Loopie-20B-A2B is pretrained on only 3.5T tokens, compared to 25T for Nemotron 3 Nano and Nemotron Cascade 2, and 36T for Qwen3-30B-A3B. On MMLU, Loopie scores higher than both Nemotron models and is competitive with GPT-OSS-20B-A2B, despite the token budget disparity. Loopie substantially outperforms the Nemotron models on BBH and IFEval, with double-digit gains on BBH and a lead of over 5 points on IFEval. Across the seven knowledge and general-capability benchmarks, it exceeds at least one of the two Nemotron models on every task and outperforms each on five of them. In post-training reasoning, Loopie ranks second among listed models on AIME 2024 and AMC, and ties Qwen3-30B-A3B on AIME 2025.

Layer-loop designs with halved layers and R=2 recurrence maintain compute parity while halving activation memory, enabling up to 1.65× the compute proxy at only 69% of the reference memory. Supervised pre-training (SPT) applies loss only to supervised targets using large batches and long sequences, jointly improving pre-training and reasoning metrics while avoiding multi-epoch overfitting. Loopie-20B-A2B, combining these advances, outperforms similarly sized MoE models on knowledge and reasoning benchmarks with less than one-seventh of their pretraining tokens, showcasing substantial token efficiency.