One-sentence Summary

CodeMidas, proposed by researchers from LLM Core, Xiaomi, Peking University, and other institutions, is an agentic pipeline that turns implemented functionality in existing codebases into executable RL environments using source code as its only task-specific input, allocating agentic compute to behavioral specification, execution-grounded test construction, and validation through execution checks and repeated solution rollouts; training MiMo-V2.5 with GRPO on the resulting 5,545 tasks from 3,185 open-source codebases spanning 23 programming languages and 15 technical domains improves performance on all five diverse benchmarks, including issue repair (DeepSWE +11.7%), whole-program construction (ProgramBench +17%), and terminal work (Terminal-Bench v2.1 +8.5%), with ablations showing that increasing the number of high-quality training tasks improves performance and trajectory analysis showing increased codebase exploration and more diverse self-verification.

Key Contributions

• CodeMidas is an agentic pipeline that turns implemented functionality in existing codebases into executable reinforcement learning environments using source code as its only task-specific input. It allocates agentic compute to exploring implemented functionality, formulating behavioral specifications, constructing execution-grounded tests, and filtering candidate tasks through execution checks and repeated solution rollouts.
• The resulting dataset has 5,545 training tasks from 3,185 open-source codebases spanning 23 programming languages and 15 technical domains.
• Training MiMo-V2.5 with GRPO on these tasks improves performance on all five diverse benchmarks, covering issue repair (DeepSWE +11.7%), whole-program construction (ProgramBench +17%), and terminal work (Terminal-Bench v2.1 +8.5%). Ablations show that increasing high-quality training tasks improves performance, and trajectory analysis shows more codebase exploration and more diverse self-verification.

Introduction

Large language models are increasingly capable of agentic coding, but reinforcement learning for software engineering depends on diverse tasks and reliable, execution-based rewards. Earlier pipelines construct environments from issues, pull requests, commits, existing tests, or documentation, which ties training coverage to those development artifacts and limits scalability. The authors introduce CodeMidas, an agentic pipeline that uses source code as its only task-specific input: it identifies implemented functionality, formulates behavioral task statements, adapts codebases into development starting points, and synthesizes tests grounded in execution of the original code with consistency checks and post-rollout filtering. Using CodeMidas, they build 5,545 verifiable tasks from 3,185 open-source codebases spanning 23 languages and 15 domains. Training MiMo-V2.5 on these tasks improves external benchmark performance, including raising DeepSWE pass rate from 10.0% to 21.7% and Terminal-Bench v2.1 pass rate from 63.7% to 72.2%, showing that implemented functionality can be converted into effective RL training signals across diverse software tasks.

Dataset

Dataset composition and scale

• The authors organize tasks around solver-visible environments that include a starting codebase, a task statement, a reference solution, and a verifier.
• After filtering, the final dataset contains 5,545 tasks from 3,185 codebases across 23 programming languages and 15 technical domains.
• The most represented languages are Python (21.4%), TypeScript (18.3%), Go (16.2%), C++ (12.5%), and JavaScript (11.3%).
• The largest technical domains are systems software (17.4%), web technologies (14.6%), and developer tools (13.6%). Together these three domains account for 45.6% of tasks.
• Reference solution size is measured by added or deleted source lines, including comments and blank lines. The median is 142 lines, with an interquartile range of 66 to 305 lines.
• In 65.9% of tasks, the reference patch touches at least two source files.

Post-rollout filtering details

• Leakage filtering: an adversarial agent searches the full solver-visible environment, including compiled artifacts, caches, files left by construction agents, and installed copies of the target project. It logs commands and outputs. A separate review rejects tasks if leaked material can bypass the intended implementation work.
• Verifier agreement filtering: a coding agent tries each task four times. A reviewing agent examines rollout trajectories, submitted code, test outputs, the task statement, the verifier, and the reference solution. It flags false positives, where an incorrect implementation passes tests, and false negatives, where a correct implementation fails. Tasks with identified verifier defects are rejected.
• Rollout outcome filtering: a frontier model makes several attempts per task, scored by the verifier. The authors keep only tasks with both successful and failed attempts under this model and budget. All-pass and all-fail tasks are excluded.

How the data is used

• The filtering is performed before RL training, so the retained 5,545 tasks form the training set.
• The verifier, reference solution, task statement, and starting codebase are used during environment construction, filtering, and scoring of agent attempts.
• The provided text reports language, domain, and reference solution statistics over the full training set.
• No training split ratios, mixture ratios, or cropping strategy are specified in the provided sections.

Method

The authors leverage source code as the sole task-specific input to construct and filter coding reinforcement learning environments. The overall framework, as shown in the figure below, outlines a multi-stage pipeline comprising task design, test construction, execution consistency checks, and post-rollout filtering.

Task Design and Codebase Adaptation The process begins with an agent inspecting the codebase structure and build metadata to identify functionalities with public entry points and observable outcomes. The authors prioritize tasks that require reasoning across the codebase, supporting interfaces such as command-line tools, pure library functions, and stateful library APIs. For each candidate, the agent traces public entry points and shared dependencies to define the task scope. It then removes the selected core implementation and adjusts the remaining code to form a coherent starting point. The task statement and code boundaries are revised together while preserving shared components, and the original implementation is retained separately as a reference solution.

Execution-grounded Test Construction To evaluate implementations, an agent maps the behavioral requirements from the task statement to test inputs and boundary cases. It invokes public entry points in a reference copy of the codebase and records the outcomes. Tests are tailored to the interface type, using command executions for CLI tools, input-output cases for pure functions, and sequences of calls for stateful APIs. Assertions are established using reference execution for fixed outputs, while unspecified aspects are checked only against stated constraints. A reviewing agent then audits every assertion to remove unsupported restrictions, replacing them with behavioral checks. Tasks relying on private symbols without behavioral substitutes are rejected.

Environment Preparation and Execution Consistency Starting from a uniform base container image, an agent installs dependencies and prepares build and runtime resources. Cleanup procedures remove artifacts that could reveal the deleted implementation, such as compiled outputs and cached copies. To ensure execution consistency, each task is evaluated in six fresh containers: two with the starting codebase and four with the reference solution. The pipeline requires that both starting-state runs fail and all four reference runs pass, verifying the expected fail-to-pass transition and screening for unstable execution outcomes.

Post-rollout Environment Filtering Before RL training, the authors apply further filtering using agent rollouts to identify exploitable leakage, verifier disagreements, and tasks with uniform pass or fail outcomes.

Leakage filtering involves an adversarial agent searching the solver-visible environment for residual leakage, such as compiled artifacts or cached files, to recover a solution without performing the intended development work. Tasks confirmed to have leaked material that bypasses implementation are rejected.

For agreement on agent solutions, a coding agent attempts each task multiple times. A reviewing agent examines the rollout trajectories, submitted code, and test outputs alongside the task statement and reference solution to detect mismatches. It flags false positives where incorrect implementations pass tests and false negatives where correct implementations fail tests, rejecting tasks with verifier defects.

Rollout outcome filtering uses a frontier model to make several attempts per task. The authors retain only tasks that exhibit both successful and failed attempts under this model, discarding those with all-pass or all-fail outcomes that may indicate weak tests or missing requirements.

Experiment

The evaluation trains MiMo-V2.5 with GRPO on 5,545 CodeMidas tasks and tests performance on five external benchmarks plus a held-out CodeMidas validation set. Reinforcement learning improves outcomes across all external benchmarks, supporting the use of source-derived functionality tasks for diverse software work. Ablations show that larger curated data pools improve results and that the high-quality 5k set outperforms a larger unfiltered 8k sample, underscoring the value of environment reliability and filtering. Behavioral analysis finds that training increases codebase exploration, code drafting, and self-verification, with self-verification associated with higher pass rates and these behavioral changes generalizing across benchmarks.

Representative coding environment pipelines differ mainly in their reliance on existing tests and development history. Most pipelines require existing tests, while newer methods are more likely to relax issue and pull request requirements, and written descriptions are generally optional. Language support is uneven, with most pipelines handling one language and only a few handling many. Almost all listed pipelines require existing tests; only one is shown as fully test-free and another as partially test-free. Issue and pull request inputs are required by only some pipelines, with several newer pipelines able to work without them. Commit history remains a common input requirement, though a few pipelines can avoid it. Written descriptions are optional for most pipelines and represent a less common constraint than tests or development history. Language coverage is concentrated: most pipelines support a single language, while broad multilingual coverage is limited to a few systems. CodeMidas is described as using source code as its only task-specific input, deriving behavioral statements and execution-grounded tests beyond the coverage of development records, documentation, and existing tests.

Reinforcement learning produced clear behavioral shifts: agents explored the codebase more before editing, increased the share of edited code already present in prior reasoning, and issued more diverse self-verification commands. Drafting showed the largest relative gain, while verification diversity increased more modestly. These changes point to more deliberate exploration and stronger consistency between reasoning and code edits. Codebase exploration before the first edit increased from early to late training, reflecting more pre-edit investigation. The drafting ratio increased substantially, indicating that written code more often appeared in preceding reasoning. Distinct post-edit self-verification commands increased, showing broader verification behavior after edits. Rollouts with self-verification had a higher mean pass rate than those without, while exploration and drafting splits showed smaller and more uncertain pass-rate differences.

From early to late checkpoints, codebase exploration increased across all evaluated held-out benchmarks. Code drafting ratios rose where measured, and self-verification diversity increased with benchmark-dependent variation. Interaction length grew for issue repair and terminal tasks but shrank for whole-program construction, where greater exploration coincided with shorter interactions. Codebase exploration increased across SWE-bench Pro, ProgramBench, and Terminal-Bench v2.1. Code drafting ratios increased on SWE-bench Pro and ProgramBench; no ratio was reported for Terminal-Bench v2.1. Self-verification diversity rose across benchmarks, with the magnitude varying by evaluation. Interaction length increased for SWE-bench Pro and Terminal-Bench v2.1 but decreased for ProgramBench.

The experiments compare representative coding environment pipelines and examine how reinforcement learning changes agent behavior. The pipeline analysis shows that most systems still depend on existing tests and development history, while newer methods relax issue and pull request requirements; written descriptions are often optional, multilingual support is limited, and CodeMidas uses source code alone to derive behavioral statements and execution-grounded tests. Reinforcement learning leads to more deliberate codebase exploration, stronger consistency between reasoning and edits, and more diverse self-verification commands, with self-verification associated with higher pass rates. Across held-out benchmarks these behavioral shifts persist from early to late checkpoints, although interaction length increases for issue repair and terminal tasks but decreases for whole-program construction.