FEA-Bench is a benchmark dataset released by Microsoft in 2025 for evaluating repository-level code generation capabilities, designed to measure the performance of large language models in repository-level incremental code development.

The dataset contains 1,401 task instances sourced from 83 GitHub repositories, derived from pull requests aimed at implementing new features (Feature Implementation). Each task instance includes repository information, base commit hash, pull request number, and unit test status information (FAIL_TO_PASS and PASS_TO_PASS), primarily used to evaluate the model's ability to solve complex software engineering tasks. It is not suitable for model training to prevent data contamination.

Dataset Composition

• instance_id: Identifier for the task instance.
• pull_number: Number of the original GitHub pull request.
• repo: Source repository name (in the format owner/repo).
• version: Repository version information.
• base_commit: SHA256 hash of the base commit.
• environment_setup_commit: Hash of the environment setup commit.
• created_at: Task creation time.
• FAIL_TO_PASS: String sequence containing failing test cases that need to be fixed.
• PASS_TO_PASS: String sequence containing test cases that must continue to pass.