One-sentence Summary

The authors from Tsinghua University and Alibaba et al. introduce EpiCoder, a feature tree-based synthesis framework that constructs hierarchical code features to enhance the diversity and complexity of generated code, enabling precise control from function-level to multi-file scenarios, and fine-tuned base models achieve state-of-the-art performance across benchmarks, with significant potential for repository-level data synthesis.

Key Contributions

• The paper introduces a feature tree-based code synthesis framework that organizes code into hierarchical features derived from high-level abstractions, enabling controllable generation of instruction data with adjustable complexity spanning function-level to multi-file scenarios.
• The framework is used to synthesize 433,000 instruction data samples for fine-tuning, producing the EpiCoder model series; EpiCoder-Qwen-7B achieves state-of-the-art results among comparably sized models on multiple function-level and file-level benchmarks, including a 7.4% improvement over Qwen2.5-Coder-7B-Instruct on BigCodeBench-Hard.
• Further analysis of the synthesized data highlights its advantages in complexity and diversity from a feature-based perspective, and empirical evidence demonstrates strong potential for scaling the framework to repository-level code synthesis.

Introduction

Large language models (LLMs) show strong capability in code understanding and generation, yet their latent code knowledge remains underutilized unless fine-tuned on high quality instruction data, which is a critical step for aligning models with user intent and improving real-world usability. Prior methods for synthesizing such instruction data mostly rely on code snippets as seed data, but snippets capture only specific functionalities and miss the full range of programming constructs, patterns, and interactions common in real applications. Their rigidity also makes it difficult to rearrange or recombine snippets into new combinations, limiting the diversity and complexity of the generated data. To overcome these constraints, the authors propose a feature tree based code data synthesis framework inspired by Abstract Syntax Trees (ASTs). The framework extracts semantic code features such as variable types, function structures, and control flow, then uses hierarchical clustering to build tree structures that capture relationships between code elements. These trees guide the LLM to extract structures from raw code, and the authors expand them iteratively in breadth and depth to enhance feature diversity. This design enables controllable complexity by adjusting subtree depth and breadth, and targeted learning by tuning sampling probabilities for underrepresented knowledge areas. The authors validate the approach by training Qwen2.5-Coder-7B-Base and DeepSeek-Coder-6.7B-Base models, yielding the EpiCoder series, which achieves state-of-the-art performance on function-level and file-level benchmarks among similarly sized models.

Dataset

The authors build a dataset by extracting hierarchical feature trees from code, using a pipeline that combines public code corpora, embedding-based selection, and LLM-driven structuring.

• Primary source: Seed data is drawn from The Stack v2, a large-scale public dataset commonly used for pre-training code LLMs. To ensure diversity and coverage, the authors apply the KCenterGreedy algorithm on code embeddings encoded by roberta-large-v1, selecting a core set of representative samples.
• Tree demonstration construction: A powerful LLM, specifically GPT-4o, is used to extract features from the seed data. Because the initial prompt heavily influences output quality, the authors propose an iterative method to refine the demonstration tree structure inside the prompt. This happens in two steps:
  • Feature pre-extraction: with a draft prompt, the LLM extracts an initial set of feature keywords from the seed data.
  • Iterative demonstration generation: the LLM performs hierarchical clustering on subsets of the feature set, producing a tree structure that captures relationships among features. The tree is refined over multiple iterations to ensure a well-organized hierarchy.
• Feature tree extraction: Using the refined demonstration, the LLM extracts a tree-structured feature representation for each code snippet. These individual trees are then merged into a unified structure that consolidates features across all samples. During merging, the authors record the frequency of each node, which reflects the distribution of features in the seed data.
• Use in the model: Since the seed data derives from the pre-training corpus, the recorded node frequencies serve as an approximate measure of knowledge distribution within the pre-trained model. This frequency information is used to guide how the dataset is applied, for example in determining mixture ratios or weighting during training. The authors do not just use the raw code, they transform it into a tree-based semantic representation that captures deeper relationships than a plain syntax tree.

Method

The authors present a feature tree-based code generation framework designed to produce diverse and complex code instruction data. The framework operates through a three-stage pipeline: feature tree extraction, feature tree evolution, and feature tree-based code generation. Refer to the framework diagram for a visual overview of these components.

The process begins with Feature Tree Extraction. To ensure data diversity, the authors collect seed data from The Stack v2 and apply the KCenterGreedy algorithm to select a core set of samples based on code embeddings. An LLM is then utilized to extract a hierarchical representation of code elements. To guide the LLM effectively, a tree demonstration is constructed iteratively. This involves pre-extracting an initial set of features, performing hierarchical clustering to organize them into a tree structure, and refining the prompt based on this structure. Once the demonstration is optimized, the LLM extracts feature trees from the seed data. These individual trees are merged into a comprehensive structure, and the frequency of each node is recorded to reflect the distribution of features.

Next, the framework employs Feature Tree Evolution to overcome limitations in feature diversity and quantity. At each iteration, a subtree is sampled from the full feature tree. This subtree is evolved by the LLM along two dimensions: depth and breadth. Evolving in depth involves adding finer-grained child nodes to existing nodes, while evolving in breadth adds sibling nodes at the same hierarchical level. These newly evolved subtrees are merged back into the overall structure. For newly generated features, the frequency is estimated as the average frequency of their siblings, ensuring seamless integration into the broader feature distribution.

Finally, the Feature Tree-Based Code Generation module utilizes the evolved tree to synthesize code. To prevent the model from focusing too heavily on high-frequency but simple features, the probability distribution of a node's child features is reweighted using a temperature parameter $t$t. The adjusted probability $p'_i$pi′​ for a child feature $i$i is calculated as:

where $p_i$pi​ is the normalized original frequency and $C$C is the set of child nodes. Subtrees are then sampled based on these adjusted probabilities. The LLM uses the sampled subtree to generate a task, the corresponding code, and the execution environment. The generated code can range from single functions to multi-file projects. To ensure quality, the process includes an iterative refinement step where test files are generated and executed in an isolated environment. Feedback from the execution results is used to guide the LLM in refining the solution code.

Experiment

The experiments evaluate EpiCoder models trained on synthetic data generated via a feature tree approach, covering function-level, file-level, and potential repository-level code generation. Function-level benchmarks show state-of-the-art average performance among same-sized models, while a custom XFileDep benchmark demonstrates strong file-level generation with cross-file dependency handling. Further analysis reveals that the synthetic data exceeds existing datasets in code complexity and feature diversity, and comparisons under matched data sizes confirm the quality and generalization advantage of the evolutionary data enhancement strategy.

The table compares Pass@1 accuracy across multiple coding benchmarks, including HumanEval, MBPP, and BigCodeBench, for both closed-source and open-source LLMs. EpiCoder-Qwen-7B achieves state-of-the-art average performance among same-size models, demonstrating the effectiveness of feature tree-based synthetic data for function-level code generation. The results also highlight that many models show lower performance on harder subsets like BigCodeBench-Hard compared to full subsets. EpiCoder-Qwen-7B outperforms other models of the same size on average across the evaluated benchmarks. Closed-source models like GPT-4-Turbo and Claude-3.5-Sonnet show strong results but are not consistently superior to the best open-source models. All models score notably lower on the Hard subset of BigCodeBench than on the Full subset, indicating increased task difficulty. The feature tree-based synthetic data approach contributes to improved performance on complex programming tasks.

The proposed code dataset exhibits substantially higher Halstead complexity than existing codebases, with notable gains in both unique and total operator/operand counts. File-level data further amplifies these differences, indicating more sophisticated and complex code synthesis. Function-level data surpasses the runner-up by 2.55 in unique operators and 20.99 in unique operands. File-level metrics exceed function-level values, showing even greater complexity across all Halstead dimensions. Total operator and operand counts for the proposed dataset are nearly double those of the closest baseline.

The proposed dataset exhibits higher feature diversity than existing code datasets, with average unique features of 8.53 at the function level and 8.95 at the file level, surpassing the nearest competitor by notable margins. Function-level data shows particular gains in areas like data processing, error handling, dependency relations, and user interaction, with counts two to three times higher than in other datasets. Function-level and file-level feature counts exceed all compared datasets, with the largest gains in user interaction and data processing. Error handling, dependency relations, and user interaction features appear at roughly two to three times the frequency found in existing codebases. File-level data shows even stronger diversity than function-level data, especially for user interaction and data processing features.

The proposed method generates code with substantially higher complexity than existing datasets across all four evaluated dimensions, with file-level code achieving the highest average scores. Function-level code also outperforms all baselines, showing consistent gains in error handling, modularity, dependencies, and data structure complexity. File-level code achieves the highest average complexity score, driven by large gains in modularity and data structure complexity. Function-level code improves average complexity by 32.6% over the strongest baseline, while file-level improves by 52.5%. The proposed method scores highest on all individual dimensions compared to existing datasets, including error handling and dependency management.

EpiCoder-DS-6.7B-Sample75k demonstrates superior performance over Magicoder-DS and WaveCoder-Ultra-6.7B on function-level benchmarks while using equal or less training data. It achieves higher average Pass@1 scores, with notable gains on several tasks. This suggests data quality plays a key role beyond data size. Achieves 5.4% and 3.0% higher average performance than Magicoder-DS and WaveCoder-Ultra-6.7B respectively. Outperforms baselines on HumanEval, MBPP, and BCB benchmarks despite using 75k samples versus up to 130k. Shows consistent improvements across completion and instruction tasks on multiple benchmarks.

The experiments evaluated EpiCoder's code generation against existing models and datasets. EpiCoder models achieved state-of-the-art average Pass@1 accuracy among same-size open-source models on coding benchmarks, though all models struggled on harder subsets. The proposed synthetic dataset showed substantially higher code complexity, feature diversity, and overall complexity compared to baselines, with file-level data surpassing function-level. Finally, EpiCoder outperformed baselines on function-level benchmarks while using less training data, indicating that data quality, not just quantity, drives performance.