One-sentence Summary

Researchers at NVIDIA introduce OpenCodeInstruct, a large-scale instruction tuning dataset of 5 million diverse coding samples—each including a programming question, solution, test cases, execution feedback, and LLM-generated quality assessments—and demonstrate that fine-tuning LLaMA and Qwen models ($1\text{B}+$1B+, $3\text{B}+$3B+, and $7\text{B}+$7B+) on it yields substantial performance gains across HumanEval, MBPP, Live-CodeBench, and BigCodeBench, thereby addressing the scarcity of high-quality, publicly available supervised fine-tuning data for code LLMs.

Key Contributions

• OPENCODEINSTRUCT is presented as the largest publicly available code instruction tuning dataset to date, containing 5 million diverse samples with rich metadata including unit tests, execution feedback, and LLM-generated quality assessments, along with a full pipeline covering seed curation, synthetic instruction generation, and filtering.

• Fine-tuning Llama3 and Qwen2.5-Coder base models across multiple scales (1B+, 3B+, and 7B+) with OPENCODEINSTRUCT yields substantial performance improvements over their instruction-tuned counterparts on HumanEval, MBPP, LiveCodeBench, and BigCodeBench, with as few as 500k samples already outperforming baseline instruct models.

• Extensive ablation and analysis reveal several key insights: LLM-based judgment is a more effective indicator of instruction quality than execution-based feedback, Genetic-Instruct combining Evol-Instruct and Self-Instruct outperforms either approach alone, larger seed sets improve downstream generation, both generic and algorithmic coding instructions contribute positively, and natural-language-to-code formatting significantly outperforms code-to-code prompting.

Introduction

Large language models for code are typically refined via instruction tuning, where curated instruction-response pairs align model outputs with user intent across software engineering tasks. The quality of instruction-following data is critical at this stage, yet generating such data at scale is both difficult and expensive.

Prior approaches have notable limitations. Human annotation, such as the large-scale dataset used to train Llama-3, yields high quality but is cost-prohibitive. Synthetic distillation methods like SELF-INSTRUCT and EVOL-INSTRUCT rely on in-context learning from limited external data, while OSS-INSTRUCT leverages real code snippets; however, these methods often require access to proprietary models and data. The previously largest open code instruction corpus contained only 435k examples.

The authors introduce OPENCODEINSTRUCT, the largest publicly available code instruction tuning dataset to date, with 5 million Python samples. It uses a scalable synthetic generation framework that combines SELF-INSTRUCT and EVOL-INSTRUCT, seeded by 1.43 million general coding instructions (extracted from GitHub via OSS-INSTRUCT) and 25,443 algorithmic questions from TACO. The pipeline also incorporates LLM-generated unit tests for feedback aggregation and LLM-based judgment for sample quality assessment. Fine-tuning Llama3 and Qwen2.5-Coder at scales of 1B+, 3B+, and 7B+ with OPENCODEINSTRUCT delivers substantial performance gains over their instruction-tuned counterparts on benchmarks including HumanEval, MBPP, LiveCodeBench, and BigCodeBench. Ablations reveal several insights: fine-tuning with only 500k samples already surpasses the original instruct models, LLM judgment is a more reliable quality indicator than execution feedback, a hybrid of Evol-Instruct and Self-Instruct yields the best results, larger seed sets improve downstream generation, and natural language to code formatting outperforms code-to-code prompting. The dataset will be fully open-sourced.

Dataset

The authors construct the OPENCODEINSTRUCT dataset for coding instruction tuning. The development pipeline combines two seed collections, a synthetic generation framework, and a series of filtering and enrichment steps.

• Dataset Composition and Sources:

  • The dataset is built from two initial seed sets: a large-scale generic collection and a smaller, high-quality algorithmic set.
  • The large-scale seeds are synthetic instructions generated from Python functions extracted from GitHub. The authors used the OSS-INSTRUCT algorithm and the Qwen2.5-32B-Instruct model to create 1.43 million coding instructions based on these functions. Only the instructions were kept; the generated solution code was discarded.
  • The small-scale seeds consist of 25,443 algorithmic coding questions sourced from TACO and various competitive programming platforms.

• Synthetic Instruction Generation:

  • The authors use the GENETIC-INSTRUCT framework to generate new instructions from the seeds. This framework uses mutation and crossover operations to create diverse questions.
  • Although the framework can run iteratively, the authors ran it for a single generation, producing nearly 9 million synthetic instructions.

• Data Cleaning and Decontamination:

  • The generated pool was refined using two steps. First, instructions containing Python code snippets were filtered out, as these were found to be noisy and unhelpful for code generation tasks. Second, an n-gram based decontamination method removed any overlap with evaluation benchmarks.
  • After this cleaning, approximately 5 million synthetic coding questions remained.

• Response Generation and Metadata:

  • The authors generated coding solutions for the cleaned instructions using the Qwen2.5-Coder-32B-Instruct model. To test the impact of the generator, they also used Qwen2.5-32B-Instruct and QwQ-32B-Preview for some samples.
  • As metadata, an LLM automatically assigned up to three coding skills to each response from a predefined list.

• Test Case Generation and Execution:

  • For each question-solution pair, the authors generated ten assertion-style unit tests using Qwen2.5-Coder-32B-Instruct.
  • All solutions were executed against their corresponding unit tests. The pass/fail results and the pass rate for each solution were included as metadata.

• Response Quality Assessment:

  • The authors used an LLM-as-a-judge approach with Qwen2.5-Coder-32B-Instruct to score each solution on requirement conformance, logical correctness, and edge case consideration. The assessment scores and justifications are included as metadata.

Method

The authors present a comprehensive framework for generating large-scale coding instruction tuning data. The pipeline begins with the creation of initial seed collections to ensure diversity and coverage. Two distinct sets of seeds are utilized in parallel: a large-scale generic collection and a small-scale algorithmic set. The generic seeds are derived from Python functions extracted from GitHub, processed through a rigorous filtering pipeline involving type checking and deduplication. These functions serve as the basis for generating diverse coding instructions. The algorithmic seeds consist of high-quality questions from the TACO dataset, focusing on competitive programming scenarios to enrich the dataset with complex data structure and algorithmic problems.

Refer to the framework diagram for an overview of the development stages.

Following the seed creation, the framework employs a scalable synthetic data generation methodology to produce instructions and their corresponding responses. This process involves evolutionary operations such as mutation and crossover, where a Large Language Model generates new instructions based on specific tasks or combines existing instructions to create diverse variants. To ensure data quality, a cleaning and decontamination step is applied. This involves filtering out instructions containing Python code snippets, which are often noisy, and removing any overlap with evaluation benchmarks using an n-gram-based method.

Once the instructions are finalized, the pipeline proceeds to response generation. The authors prompt a specialized coding model to generate high-quality code solutions for the instructions. Additionally, metadata in the form of coding skills is automatically generated to categorize the solutions. To further enhance the dataset's utility, particularly for reinforcement learning with execution feedback, the framework generates assertion-style unit tests for each question-solution pair. These tests are created by a model and then executed against the generated solutions. The execution engine captures the results, including pass rates and error types. The distribution of test outcomes reveals that a significant portion of solutions pass all tests, while others fail due to assertion errors or other execution issues, as depicted in the test case pass/fail distribution chart.

The execution results are integrated as metadata, providing valuable feedback signals for training. This end-to-end pipeline ensures the generation of a robust, diverse, and high-quality dataset suitable for instruction tuning in coding tasks.

Experiment

The evaluation fine-tunes Llama3 and Qwen2.5-Coder models with OPENCODEINSTRUCT and compares them against instruction-tuned baselines on code generation benchmarks, showing consistent improvements, especially for general purpose models. Ablation studies reveal that LLM based verification outperforms unit test filtering, larger synthetic data scales improve performance, both instruction generation algorithms are complementary, diverse seed populations enhance quality, and natural language to code formatting is more effective than code to code. Additionally, using a stronger generator model yields better downstream performance, and OPENCODEINSTRUCT outperforms an equivalent size dataset based on existing code snippets, demonstrating its higher per sample quality.

The table compares the sample sizes of various publicly available code-instruction tuning datasets, showing a wide range from 20,000 to 435,000 samples. OPENCODEINSTRUCT is positioned as the largest such dataset, with 5 million synthetic samples generated from a diverse seed set of Python functions and algorithmic questions. The dataset leverages a scalable generation framework combining self-instruct and evol-instruct methods, and is reported to yield higher quality instructions on a per-sample basis compared to existing alternatives. OPENCODEINSTRUCT contains 5 million samples, far exceeding all other listed datasets, the largest of which has 435,000 samples. The dataset uses a much larger and more diverse seed set, including 1.43 million Python functions and 25,443 algorithmic questions, unlike limited seed-based approaches. Evaluation results indicate that fine-tuning with OPENCODEINSTRUCT improves code generation performance over using OSS-INSTRUCT samples by several points across benchmarks.

The table shows the distribution of algorithmic questions across multiple competitive programming platforms in the TACO dataset. CodeForces contributes the largest share, followed by CodeChef and GeeksForGeeks, indicating broad platform diversity. This distribution supports the use of TACO as a diverse seed set for generating synthetic coding instructions. CodeForces is the most represented platform, providing the highest number of questions. The dataset spans multiple platforms, including AIZU, AtCoder, Codewars, and HackerEarth, ensuring varied algorithmic coverage. The multi-platform distribution enhances the diversity of the seed set for instruction tuning.

Fine-tuning Llama3 and Qwen2.5-Coder models with OPENCODEINSTRUCT improves performance over their instruction-tuned baselines across function-level and library-based coding benchmarks. Gains are substantial for Qwen2.5-Coder, while smaller Llama models show limited improvement on the more complex LiveCodeBench tasks. The OPENCODEINSTRUCT-tuned Llama model shows large gains on HumanEval and MBPP but only a slight increase on LiveCodeBench. The Qwen2.5-Coder model fine-tuned with OPENCODEINSTRUCT outperforms its baseline on every listed benchmark, with a particularly strong jump on LiveCodeBench. The finetuned Qwen model also surpasses the OpenCoder-1.5B-Instruct baseline, indicating competitive performance from the new dataset.

Filtering synthetic training data based on unit-test execution feedback and LLM judgment scores improves finetuning outcomes for Qwen2.5-Coder-7B. Selecting samples that pass unit tests or receive high LLM judgment scores yields better code generation performance than random selection, while excluding failing samples degrades results.

Ablation results show that both instruction generation algorithms contribute complementary strengths, with Evol-Instruct generally performing better on the tested benchmarks for stronger base models. Using a larger, more diverse seed population yields substantial gains, and combining algorithmic with generic instructions improves overall performance. Additionally, NL-to-Code formatting consistently outperforms Code-to-Code formatting across all evaluated benchmarks. Evol-Instruct tends to outperform Self-Instruct on several benchmarks, especially for the Qwen-based model. Large-scale seed sets with diverse algorithmic questions lead to marked performance improvements over smaller seed sets. Mixing algorithmic and generic seed instructions yields superior results compared to either alone, despite comparable performance on some individual benchmarks. NL-to-Code instructions significantly beat Code-to-Code instructions across all tested code generation benchmarks.

Experiments introduce OPENCODEINSTRUCT, a 5 million sample synthetic code instruction dataset built from a diverse seed set including 1.43 million Python functions and algorithmic questions sourced from platforms like CodeForces. Fine-tuning language models, particularly Qwen2.5-Coder, with this dataset improves code generation performance over baselines, with notable gains on benchmarks like LiveCodeBench. Filtering synthetic samples based on unit test execution and LLM judgment scores further boosts outcomes, while ablations show that Evol-Instruct, larger seed sets, mixing algorithmic and generic instructions, and NL-to-Code formatting each contribute to stronger results.