One-sentence Summary

NVIDIA researchers propose AceMath, a suite of instruction-tuned math and reward models, employing an SFT process that first achieves competitive general-domain performance before targeting math with curated prompts and synthetic responses; AceMath-72B-Instruct outperforms GPT-4o and Claude-3.5 Sonnet, while AceMath-72B-RM, developed using the new AceMath-RewardBench, surpasses state-of-the-art reward models and attains the highest average $\mathrm{rm@8}$rm@8 score across math reasoning benchmarks.

Key Contributions

• The paper introduces AceMath-Instruct, a suite of math-tuned models trained with a two-stage supervised fine-tuning process that first builds general-domain instruction-following ability, then applies targeted math SFT with curated prompts and synthetic responses. Results show the 72B variant greatly outperforms Qwen2.5-Math-72B-Instruct, GPT-4o, and Claude-3.5 Sonnet, while the 7B model surpasses Qwen2.5-Math-7B-Instruct and nearly matches the 10x larger 72B counterpart.

• The paper constructs AceMath-RewardBench, a comprehensive benchmark for evaluating math reward models across a diverse range of datasets and difficulty levels.

• The paper develops AceMath-RM, a math-specialized outcome reward model built with a systematic training approach that samples diverse model responses to improve robustness against style biases. The 72B reward model consistently outperforms state-of-the-art reward models such as Qwen2.5-Math-RM-72B and Skywork-o1-Open-PRM-Qwen-2.5-7B, and combining it with AceMath-72B-Instruct achieves the highest average rm@8 score across math reasoning benchmarks.

Introduction

Mathematical reasoning is a fundamental and verifiable capability for large language models, making it a reliable benchmark for evaluating complex, multi-step logical reasoning. Prior math-specialized models achieve gains through continued pre-training on massive math corpora and supervised fine-tuning (SFT) on prompt-response pairs, but they still lag behind much larger general models at smaller scales, and existing reward models often suffer from stylistic biases. The authors introduce AceMath, a series of frontier-class math instruct and reward models, using a two-stage SFT approach that first trains on general domains before math-specific fine-tuning. They also conduct a systematic study of reward model training, focusing on positive-negative pair construction, training objectives, and debiasing, which yields a verifier that outperforms state-of-the-art counterparts. AceMath-7B-Instruct surpasses Qwen2.5-Math-7B-Instruct and approaches the performance of a 10x larger model, while AceMath-72B-RM sets new standards, and the authors open source both the model weights and complete training data plus a new reward benchmark.

Dataset

The authors construct a supervised fine-tuning (SFT) dataset divided into general and math-specific components, alongside a separate dataset for reward model training. Here is a breakdown of the dataset composition, processing, and usage.

General SFT Data

• Sources: Prompts are collected from a diverse range of open-source datasets. General domain sources include ShareGPT, SlimOrca, EvolInstruct, GPTeacher, AlpacaGPT4, and UltraInteract. Coding domain sources include Magicoder, Wizard-Coder, GlaiveCodeAssistant, and CodeSFT. Math sources include NuminaMath, OrcaMathWordProblems, MathInstruct, MetaMathQA, and new synthetic data.
• Prompt Processing: After collection, the authors perform data deduplication to remove identical prompts (based on lowercase comparison). The remaining prompt set is retained unfiltered to preserve diversity.
• Response Construction: Original open-source responses are discarded to ensure consistent quality and format. Instead, GPT-4o-mini generates new responses for all coding and general domain prompts using greedy decoding. Math prompts use a separate process.
• Size: This process yields around 1.2 million coding samples (0.67 billion tokens) and 0.7 million general domain samples (0.55 billion tokens). Additionally, 1.2 million samples from the math SFT dataset are used here.

Math SFT Data

• Initial Prompts: The authors collect over 1.3 million initial prompts from open-source datasets (NuminaMath, OrcaMathWordProblems, MathInstruct, MetaMathQA) after deduplication. These cover a wide range of difficulties.
• Synthetic Prompts: To increase diversity, they generate approximately one million additional synthetic prompts using GPT-4o-mini. The process uses NuminaMath as seed data and applies in-breadth and in-depth evolution strategies. Low-quality prompts are filtered out, including those generated by a constraint-adding evolution type, prompts exceeding 300 words, and 500K other low-quality samples.
• Response Construction: Qwen2.5-Math-72B-Instruct generates responses for the over 2.3 million total prompts. The model is instructed to reason step by step and provide the final answer in a boxed format. Responses are filtered for format consistency, length (excluding those over 2,500 words), and repetitive patterns. This results in about 2.3 million math SFT samples (1.83 billion tokens).
• High-Quality Subset: To improve accuracy, the authors cross-check answers. They generate two responses per prompt with GPT-4o-mini and only consider answers consistent across both. They then select responses where this answer matches the one from Qwen2.5-Math-72B-Instruct, yielding a high-quality subset of 800K samples.

Data Decontamination

• Process: To prevent test set contamination, the authors apply a filtering process. For math prompts, they normalize text and remove irrelevant punctuation, then filter out prompts with a 13-gram overlap with test data or a longest common subsequence exceeding 60% of the prompt's length. For non-math prompts, a 13-gram overlap with test samples is sufficient for removal.

How the Data is Used

• General SFT: The general SFT model is trained first to establish strong instruction-following and broad question-answering skills. It uses a mix of 1.2 million coding samples, 0.7 million general samples, and 1.2 million math samples (0.95 billion tokens).
• Math SFT: The math SFT stage starts from the general SFT model. The training blend consists of all NuminaMath samples, a subset of the million synthetic prompts, and the 800K cross-checked high-quality samples. After duplicate removal, this totals about 1.6 million samples (1.29 billion tokens). The authors find this blend performs better than using all 2.3 million math samples.

Reward Model Training Data

• Initial Construction: The authors use a 350K subset of the math SFT data, using the prompts and GPT-4o-mini generated answers as reference labels. They sample four responses per problem from 14 different LLMs, including various Llama, DeepSeek, Mistral, Gemma, and Qwen models.
• Scoring and Selection: The responses are annotated as correct or incorrect against reference labels. To reduce false negatives, the reward model Qwen2.5-Math-RM-72B ranks the candidates. The authors use score-sorted sampling, selecting from the top-k positive and bottom-k negative candidates, with k set to 14. They sample six responses per problem with a balanced number of positive and negative examples and filter out problems where all responses are correct or incorrect.
• Addressing Biases: To handle stylistic biases, the authors use few-shot prompting to generate short reasoning paths for 2,000 multiple-choice problems. They also sample 30,000 problems and generate responses using AceMath-Instruct checkpoints to create positive and negative pairs.
• Final Dataset: The final reward model training dataset consists of 356K problems, each with six responses.

Reward Model Evaluation Benchmark (AceMath-RewardBench)

• Composition: The benchmark is built to evaluate math reward models. It uses 7 datasets and includes 64 candidate responses per problem, sampled from 8 different LLMs.
• Evaluation Metric: The primary metric is rm@8, which involves randomly sampling 8 responses from the 64 candidates and averaging the accuracy over 100 random seeds for statistical reliability. The benchmark uses the MATH500 subset for evaluation.

Method

The authors propose a comprehensive training pipeline that begins with a two-stage supervised fine-tuning (SFT) process to establish a robust foundation for subsequent math-specific training. The methodology encompasses data curation, a staged training strategy, and the development of a reward model.

General SFT Data Curation and Strategy To build a general SFT model capable of following instructions and answering diverse questions, the authors collect prompts from a wide array of open-source datasets categorized into general, coding, and math domains. To address inconsistencies in original responses, the system generates new responses using GPT-4o-mini via greedy decoding, ensuring a uniform format and high quality. This process yields approximately 1.2 million coding samples (0.67 billion tokens) and 0.7 million general domain samples (0.55 billion tokens).

Refer to the data distribution overview below:

The total constructed dataset comprises approximately 2.3 million math SFT samples (1.83 billion tokens), 1.2 million coding SFT samples (0.67 billion tokens), and 0.7 million other samples (0.55 billion tokens). A strategic portion of the math data, specifically 1.2 million samples (0.95 billion tokens), is repurposed for general SFT to enhance the model's instruction-following capabilities before specialized math training.

The training process is divided into two distinct stages. In Stage 1, the model is trained on a large dataset focused primarily on coding and math tasks to build a strong foundational capability. Stage 2 expands the model's scope by incorporating a balanced mix of coding, math, and general SFT data. For Stage 2, the authors select only cross-checked high-quality math samples where the final answers from GPT-4o-mini and Qwen2.5-Math-72B-Instruct align, ensuring the integration of diverse and reliable data.

Math SFT Strategy The math-specific SFT phase utilizes the general SFT model as its starting point. The dataset for this phase is constructed by merging samples from NuminaMath, a subset of synthetic prompts, and 800K math SFT samples that were cross-checked between GPT-4o-mini and Qwen2.5-Math-72B-Instruct. After removing duplicate prompts, the final dataset contains 1.6 million samples. This curated blend is found to yield better performance than utilizing the entire pool of math SFT data.

Training Details All SFT models are optimized using the AdamW optimizer. The learning rate is set to $5 \times 10^{-6}$5×10−6 for general SFT and $3 \times 10^{-6}$3×10−6 for math SFT. A global batch size of 128 is employed across all model sizes, with the exception of the 72B model, which uses a batch size of 256. Training proceeds for one epoch with a maximum sequence length of 4096.

Reward Model Training To select accurate solutions and superior reasoning paths, a math reward model is trained using an outcome reward approach. The architecture initializes the backbone from the supervised fine-tuned model and adds a linear layer at the top to project the last token representation into a scalar value.

The training data synthesis involves generating responses from a set of 14 different LLMs and annotating them as correct or incorrect against reference labels. To mitigate noise from heuristic evaluation tools, the authors employ a score-sorted sampling strategy, ranking candidates using an existing reward model and sampling from the top-k positive and bottom-k negative responses. This results in a dataset of 356K problems, each paired with six responses.

The model is trained using a list-wise Bradley-Terry loss to maximize the margin between correct and incorrect responses:

Here, $r _ { \theta } ( x , y )$rθ​(x,y) denotes the output score of the reward model for problem $x$x and response $y$y. This loss function is designed to optimize the model's discriminative ability between positive and negative candidate scores.

Experiment

The evaluation setup covers general SFT benchmarks (coding, math, and knowledge tasks) and extensive mathematical benchmarks ranging from grade school to Olympiad level, with both models and a new reward benchmark (AceMath-RewardBench) used for assessment. AceInstruct models trained with a two-stage strategy consistently outperform single-stage counterparts and their instruct baselines, with the largest gains on weaker base models and coding/math tasks. AceMath-Instruct models significantly surpass Qwen2.5-Math-Instruct at all scales, with the 7B variant competitive with much larger models such as Llama3.1-405B and GPT-4o, while ablations confirm that general SFT before math SFT, carefully selected synthetic data, and diverse model-generated training responses are all critical for performance. Reward modeling results show that AceMath-RM achieves state-of-the-art accuracy, especially on the harder RewardMATH benchmark, and that increasing model size provides greater benefits than adding more data, particularly on challenging college-level and competition datasets.

AceMath models achieve competitive or state-of-the-art results across diverse math reasoning benchmarks, with the reward model based rm@8 selection consistently improving accuracy over direct generation. The 72B model generally leads, while smaller variants still perform strongly against much larger proprietary models. AceMath-72B-Instruct achieves the highest or near-highest scores on GSM8K, MATH, and MMLU STEM among all compared models, including GPT-4o and Claude 3.5 Sonnet. Using AceMath-72B-RM for best-of-8 selection (rm@8) improves accuracy for every AceMath model size across all seven benchmarks, with gains often exceeding 3 points. The 7B and 1.5B AceMath models with rm@8 outperform proprietary models on several benchmarks, such as Minerva Math and College Math, despite their smaller size.

The proposed two-stage supervised fine-tuning strategy improves performance across diverse base models, with the largest gains observed on coding and math benchmarks. AceInstruct models consistently outperform their corresponding instruct baselines, and they achieve comparable results to more advanced baselines on stronger starting models. Average score improvements exceed 4% over baselines for DeepSeek-Coder, Llama3.1, and Qwen2.5 models. DeepSeek-Coder-based AceInstruct shows particularly large gains, with about 10% or more average improvement on coding and math tasks. On Qwen2.5-7B and Qwen2.5-72B, AceInstruct delivers performance comparable to the corresponding instruct models.

Ablation experiments compare a two-stage supervised fine-tuning strategy against single-stage variants on two base model families, Qwen2.5 and Llama3.1. The two-stage AceInstruct models consistently outperform both single-stage setups across all evaluated benchmarks, with notably larger gains observed for the weaker Llama3.1 base model. The stronger Qwen2.5 base shows smaller improvements, likely because it already benefits from substantial math and coding data during pretraining. Two-stage training outperforms single-stage training on both base model families across all tasks. Llama3.1-8B gains more than 3 average points from two-stage training, while Qwen2.5-7B shows a smaller but consistent advantage. The benefit of incorporating extensive coding and math data is more pronounced for weaker base models, as stronger models may already leverage such knowledge from pretraining.

AceMath-Instruct models achieve notably higher scores than their Qwen2.5-Math-Instruct counterparts across math benchmarks, with the 72B variant exceeding the previous state of the art by a substantial margin. The 7B model also performs competitively, matching or surpassing much larger models like GPT-4o and Claude-3.5 Sonnet. AceMath-72B-Instruct outperforms Qwen2.5-Math-72B-Instruct with an average improvement of 3.68 points. AceMath-7B-Instruct rivals GPT-4o and Claude-3.5 Sonnet, and is close to the much larger Qwen2.5-Math-72B-Instruct in average score. AceMath-Instruct models consistently beat their corresponding Qwen2.5-Math-Instruct baselines at 1.5B, 7B, and 72B scales.

Ablation experiments show that using responses from either GPT-4o-mini or Qwen2.5-Math-72B-Instruct individually yields performance close to using both, indicating robustness in data construction. Increasing math SFT data quantity or using only high-quality samples does not improve results, while combining cross-checked high-quality data with diverse samples works best. Skipping general SFT before math SFT causes an average score drop of about 1%. Relying solely on GPT-4o-mini responses results in only about a 1% average score drop for one backbone, showing low dependence on powerful math experts. Using only 800K high-quality math samples or expanding to 2.3 million samples does not outperform the 1.6 million mixed set, favoring diversity over quantity or strict filtering. Omitting the general SFT step consistently lowers average scores by roughly 1%, even when starting from a math-base backbone.

Evaluation across math benchmarks shows that AceMath models, especially with best-of-8 reward model selection, achieve competitive or state-of-the-art accuracy, with the 72B variant leading. The two-stage supervised fine-tuning for AceInstruct consistently improves over single-stage baselines, particularly on coding and math tasks and for weaker base models. Ablations confirm that mixing high-quality and diverse math data is more effective than scaling quantity alone, and that a general SFT stage before math SFT is essential, as its omission drops scores by about 1%. Overall, these experiments validate the proposed training and data strategies for strong math reasoning performance.