One-sentence Summary

Researchers from Peking University and collaborators introduce DataPrep-Bench, a unified benchmark assessing LLM-driven data construction and quality evaluation via downstream utility, featuring a skill-guided Data-Construction-Skill agent that improves finance fine-tuning $\sim 20\text{ points}$∼20 points and the Distributional Alignment Score (DAS) evaluator achieving cross-model correlation $r > 0.70$r>0.70 in Math, Science, and Medical.

Key Contributions

• DataPrep-Bench, a unified benchmark, is introduced to jointly evaluate LLM-driven data construction and data quality evaluation under a shared downstream-grounded protocol covering six domains and multiple model architectures.
• The data construction track releases Data-Construction-Skill, a skill-guided agent, together with its curated raw source corpus. This agent improves the Dolly-only baseline by nearly 20 absolute points on Llama-3.1-8B Finance and is competitive with the strongest agent- and DataFlow-based methods in knowledge-extraction-dense domains.
• The data quality evaluation track releases the Distributional Alignment Score (DAS), an MMD-based evaluator, along with candidate pools and ground-truth downstream performance records. DAS attains the strongest cross-model correlation in four of six domains and is the only metric surpassing r > 0.70 simultaneously in Math, Science, and Medical.

Introduction

The quality and scale of training data critically determine LLM performance, and as naturally occurring corpora fail to meet growing demand, practitioners increasingly use LLMs and agents to synthesize supervised training sets. However, comparing data construction strategies and quality evaluation metrics is challenging because existing studies rely on inconsistent raw sources, base models, and downstream benchmarks, leaving it unclear which approaches reliably produce useful training data or predict real utility. The authors introduce DataPrep-Bench, a unified benchmark that evaluates both data construction and data quality evaluation under shared domains, models, and training protocols, enabling fair, downstream-grounded comparisons.

Dataset

The authors introduce DataPrep-Bench, a benchmark that evaluates how well LLMs prepare training data, and it provides carefully curated datasets for two tracks: Data Construction and Data Quality Evaluation. The dataset description focuses on those two components, their composition, and how they are used in the paper.

Data Construction Track – Source Corpora

• Six domains: Science, Law, Finance, Medical, Math, and General Text.
• Source material (except General Text): expert-authored, long-form books (hundreds of pages per book) from open repositories such as Wikibooks, FreeBookCentre, and Open Textbook Library. For Math, the corpus also includes Chinese university math textbooks and Chinese math competition materials. For Science, the corpus additionally contains preparatory and problem materials from International Science Olympiads (IPhO, IChO, IBO) to inject high-difficulty reasoning.
• General Text: 150 MB of text randomly sampled from the FineWeb sample-10BT subset and organized into 100 Markdown files, reflecting a web-scale distribution.
• Filtering and curation: books are hand-selected to cover major subfields (at least three books per subfield). No further heuristic filtering is applied.
• Processing: All PDF sources are converted to Markdown using MinerU, preserving section structure, tables, and formulas as faithfully as possible. Each domain forms a folder of Markdown files.
• Usage: A data construction method takes the domain folder as input and produces a supervised fine-tuning (SFT) dataset of question-answer pairs. The base model $f_0$f0​ is then fine-tuned on that constructed dataset, and the method’s score is the downstream benchmark performance on an independent, curated test set $\mathcal{T}_k$Tk​ for the same domain. No fixed size or mixture ratios are imposed; different methods may produce datasets of very different scales.

Data Quality Evaluation Track – Candidate and Proxy Datasets

• Candidate pools (one per domain): These are collections of publicly available SFT corpora mixed to include both in-domain and out-of-domain data. The mix is intentional: it creates a spread of ground-truth utilities so that metrics can be reliably correlated.
• Pool sizes: General Text has 14 candidates; specialized domains (Math, Science, Medical, Finance, Law) each have 8–10 candidates. This variation reflects the scarcity of public in-domain SFT data outside general text.
• Proxy datasets (one per domain): A high-quality in-domain SFT corpus selected as a reference distribution for metrics that compare candidate features against a proxy (e.g., DAS). Proxies are chosen for domain alignment, quality, and disjointness from both the candidate pool and the downstream test set.
  • General: Infinity-Instruct
  • Math: ODA-Math-460k
  • Science: Logics-STEM
  • Medical: ReasonMed
  • Finance: Fin-o1
  • Law: DISC-Law-SFT
• Usage: No data is constructed here. Instead, the authors compute the ground truth by fine-tuning $f_0$f0​ on each candidate dataset and measuring performance on the domain-specific test set. A quality metric must predict this downstream training utility without actually training. The proxy dataset is used only for metric computation and is never trained on.

Method

The authors propose Data-Construction-Skill, a skill-guided agentic framework designed to transform long-form domain corpora such as textbooks or technical manuals into high-quality, reusable QA-style supervision data for post-training. The method addresses a core bottleneck in this setting: when supervision must be faithfully extracted and reformulated from hundreds of pages of expert-authored content, a single generation prompt cannot reliably handle task decomposition, output consistency, validation, coverage, and resumable execution simultaneously. Data-Construction-Skill retains the flexibility of an agent responsible for planning and execution while moving the specification of what constitutes valid supervision into a reusable skill layer. This skill packages task instructions, output schemas, quality constraints, filtering rules, and auxiliary resources into a structured interface that guides the agent across multiple chunks without hard-coding the entire workflow.

The construction pipeline begins by partitioning a source book corpus $B_k$Bk​ for a given domain into semantically coherent chunks $\mathcal{C}_k = \text{Chunk}(B_k)$Ck​=Chunk(Bk​), using a structure-aware chunking procedure that preserves local completeness while keeping each unit manageable for downstream processing. For each chunk $c \in \mathcal{C}_k$c∈Ck​, the agent first filters out segments that are navigational, noisy, repetitive, or otherwise uninformative. For retained chunks, it identifies the main knowledge propositions and their local semantic relations, including definitions, rules, mechanisms, conditions, exceptions, comparisons, and causal links. Based on this analysis, the method adaptively generates up to three complementary types of QA supervision for $c$c: concept-oriented pairs $\mathcal{G}^{\text{concept}}(c)$Gconcept(c) that capture reusable atomic knowledge (categories, functions, constraints), reasoning-oriented pairs $\mathcal{G}^{\text{reason}}(c)$Greason(c) that encode concise source-grounded reasoning patterns, and case-based pairs $\mathcal{G}^{\text{case}}(c)$Gcase(c) that place the same knowledge into simple structured application scenarios. These three sets are pairwise disjoint, and the chunk-level supervision is their union:

All supervision is instantiated as instruction-response pairs. The final dataset for domain $k$k is the aggregation over all processed chunks:

A central innovation is the skill layer that regulates this process. Instead of embedding all control logic in a single prompt, the skill provides the agent with explicit, reusable specifications for sample types, output schema, filtering criteria, coverage requirements, and validation behavior. It instructs the agent to produce QA pairs that are faithful to the source, self-contained, diverse in question type and difficulty, and useful as supervised fine-tuning examples. It also defines invalid patterns: document-relative questions, hallucinated answers, duplicated samples, and questions whose answers cannot be supported by the source. This skill layer acts as an intermediate control interface between high-level construction goals and low-level agent execution, stabilizing agent behavior and enabling reusable workflows.

After generation, the authors apply a post-processing and validation phase to improve dataset usability. They remove document-relative phrasing and passage-dependent references, ensuring the resulting examples reflect general domain supervision rather than book-comprehension behavior. Malformed, duplicated, or weakly grounded samples are filtered. Chunk-level processing records are maintained to support long-document execution, coverage checking, and resumability. In this way, Data-Construction-Skill provides a robust, scalable approach to transforming exhaustive domain books into faithful and diverse QA supervision corpora.

Experiment

The Data Construction Track evaluates methods that turn raw domain corpora into supervised fine-tuning data, revealing that blindly adding synthetic domain data often degrades performance, while agent-based methods excel in reasoning-heavy domains and skill-guided extraction boosts dense knowledge-domain results but stumbles on open-ended reasoning. The Data Quality Evaluation Track measures how well metrics predict downstream utility before any training, showing that the proposed DAS metric is the only reliable predictor across multiple reasoning-intensive domains, whereas existing metrics are either narrow specialists or sign-inconsistent and thus actively misleading in practice. Together, the tracks demonstrate that downstream-grounded evaluation is crucial because surface-level quality signals frequently give wrong guidance.

The benchmark pairs each of six domains with a high-quality proxy dataset that acts as a target-distribution anchor, such as Infinity-Instruct for General Text and ODA-Math-460k for Math, while the candidate pool mixes in-domain and out-of-domain public SFT corpora to create a spread of downstream utility values that metrics must linearly track. Infinity-Instruct serves as the proxy for General Text; domain-specific proxies like ReasonMed and DISC-Law-SFT anchor Medical and Law. The candidate pools intentionally blend in-domain and out-of-domain data to avoid narrow utility ranges and trivial out-of-domain detection. General Text hosts the largest candidate pool with 14 candidates, whereas specialized domains like Finance and Law start with fewer candidates but can be extended without invalidating existing scores.

On Qwen2.5-7B, mixing domain-specific synthetic data with Dolly-15k often degrades Math performance, with every generator falling below the Dolly-only baseline. The exception is Finance, where the DataFlow-Skill pipeline yields a clear improvement over the baseline, while direct LLM generation substantially hurts performance. All synthetic-data mixes reduce the Math average relative to the Dolly-15k-only baseline, with drops ranging from about 2 to 6 points. DataFlow-Skill achieves a 64.8 Finance average, notably higher than the 57.8 baseline, whereas Claude Opus 4.6 generation causes a steep drop to 44.4.

Fine-tuning Llama-3.1-8B on Dolly-15k plus domain-specific synthetic data degrades Math and General performance compared to using Dolly-15k alone, while it substantially improves Finance results. DataFlow-based generators deliver the strongest Finance gains, with DataFlow-Skill achieving the highest average, whereas direct LLM generation yields the weakest Finance improvement. The pattern shows that adding synthetic domain data is not universally beneficial and can hurt performance in reasoning-focused domains. The Dolly-15k only baseline reaches a Math average of 11.7 and a General average of 67.1, exceeding every configuration that adds domain-specific synthetic data in these categories. In Math, all synthetic data generators cause a drop, with averages between 5.8 and 9.3 compared to the 11.7 baseline. General (MMLU-Redux) scores fall sharply from 67.1 to a range of 48.4–52.9 when any domain data is mixed in. Finance is the only domain where synthetic data helps: the baseline's 15.1 average is surpassed by all generators, with DataFlow-Skill reaching 36.5. Among the tested generators, DataFlow-based methods (DataFlow and DataFlow-Skill) lead in Finance, while the LLM-based Claude Opus generator trails with a 28.4 average. Adding domain-specific synthetic data consistently hurts Math and General, illustrating that more synthetic data is not a safe default and can regress performance in reasoning tasks.

Adding domain-specific synthetic data to the Dolly-15k instruction-tuning corpus frequently degrades Qwen2.5-7B’s downstream performance, with science benchmarks dropping below the Dolly-only baseline no matter which generator is used. While DataFlow-based methods deliver modest law gains, they cause large regressions in medical and science scores, and the strongest LegalBench result from DataFlow-Skill comes at the cost of a weaker LexGLUE score. The Dolly-only baseline achieves the best science average and competitive law results, illustrating that more synthetic data is not a safe default. The Dolly-15k only baseline yields the highest science average (27.9) and outperforms all domain-augmented variants on that cluster for Qwen2.5-7B. DataFlow lifts the law average from 74.5 to 77.2, but the science average drops sharply from 27.9 to 20.1, with a particularly steep fall on MMLU-STEM (47.5 to 37.9). DataFlow-Skill improves LegalBench to 92.0 (up from 86.9) but causes LexGLUE to drop from 62.0 to 57.4, and its science average falls to 23.4. Claude Opus 4.6 generation produces a dramatic decline on MedMCQA (27.4 to 8.1) and leaves the overall science average at 23.1, well below the baseline.

Fine-tuning Llama-3.1-8B on a mixture of Dolly-15k and domain-specific synthetic data produces uneven effects: Law and Medical benchmarks improve over the Dolly-only baseline across all generators, while Science performance consistently drops, with DataFlow-based methods collapsing the most. The results show that adding synthetic data can actively harm downstream accuracy in reasoning-heavy domains, and the choice of generation method significantly impacts outcomes. All generators improved Law and Medical averages relative to the Dolly-only baseline, with gains up to 3.6 points in Law and 15.7 points in Medical. In Science, every synthetic data condition scored below the Dolly-only baseline; DataFlow-Skill fell to an average of 3.1 versus 13.2 for Dolly-only, including zero on one benchmark. Claude Opus 4.6, a direct LLM-prompted generator, achieved the highest Medical average and the smallest Science regression, yet still underperformed Dolly-only in Science by 3.9 points. DataFlow-based pipelines collapsed on open-ended Science benchmarks like SciBench and MMLU-Pro, while performing competitively on structured Law and Medical tasks such as LegalBench and MedMCQA.

Experiments fine-tuned Qwen2.5-7B and Llama-3.1-8B on Dolly-15k mixed with domain-specific synthetic corpora produced by different generators, evaluating downstream utility across Math, Finance, Law, Medical, Science, and General text. Adding synthetic data consistently degraded performance on reasoning-heavy domains like Math and Science relative to the Dolly-only baseline, while occasional gains in Finance or Law were uneven and often carried trade-offs. The choice of generator had a pronounced impact, with pipelines performing well in one domain but collapsing in another, and simply blending more in-domain synthetic data proved unreliable as a default strategy.