One-sentence Summary

KAIST researchers propose J-ZERO, a unified Challenger–Solver–Judge coevolution framework from zero data that supports self-improvement in both verifiable and unverifiable domains, in which the Challenger and Solver co-evolve adversarially while the Judge co-adapts using production-order preference pairs rather than its own scores; the framework outperforms baselines by $4.2$4.2 points on verifiable and $8.0$8.0 points on unverifiable domains and continues improving through at least ten iterations, whereas baselines degrade after two.

Key Contributions

• J-ZERO is a zero-data self-evolving framework in which a Challenger, Solver, and Judge co-evolve, supporting self-improvement in both verifiable and unverifiable domains without external data or human feedback.
• The Judge co-adapts using preference pairs whose ordering is known in advance from how responses are produced, such as Solver responses over Challenger responses and decomposed-and-recombined answers over one-shot answers.
• At two model scales, J-ZERO outperforms prior zero-data methods by an average of 4.2 points on verifiable and 8.0 points on unverifiable domains, and it continues improving through at least ten iterations while existing approaches degrade after two.

Introduction

Self-evolving large language models reduce reliance on costly human supervision, but data-free self-play has mainly been effective in verifiable domains where objective answers provide clear reward signals. In unverifiable domains, prior approaches depend on a fixed Judge, which creates an evaluation ceiling: once the Solver surpasses the Judge’s internal preferences, further training produces no useful signal. The authors propose J-ZERO, a zero-data framework in which the Challenger, Solver, and Judge co-evolve. The Judge is trained using preference pairs derived from structural asymmetries in the loop: Solver responses are preferred over Challenger responses, and divide-and-conquer responses are preferred over one-shot responses. This enables the evaluation signal to improve over time and supports sustained self-improvement in both verifiable and unverifiable domains.

Method

The authors propose J-ZERO, a self-evolving framework that co-adapts the Judge within the self-play loop alongside the Challenger and the Solver, rather than keeping the reward model fixed throughout training. This approach mitigates the performance ceiling imposed by static reward models. Self-evolution proceeds iteratively, with each iteration comprising three distinct stages.

As shown in the figure below:

In the first stage, the Challenger learns to generate progressively more challenging tasks by minimizing the reward that the Judge assigns to the Solver responses. In the second stage, the Solver is trained to produce higher-quality responses to these difficult tasks by maximizing the Judge reward. Finally, in the third stage, the Judge is updated using the Bradley-Terry loss on preference pairs constructed within the self-play loop.

Adversarial Evolution of the Challenger and Solver The authors frame the interaction between the Challenger $C_{\theta_c}$Cθc​​, the Solver $S_{\theta_s}$Sθs​​, and the Judge $J_\phi$Jϕ​ as an asymmetric adversarial game. The Challenger samples a batch of tasks, and the Solver generates multiple responses for each task. The Judge assigns a scalar score to each task-response pair. The interaction is formulated as:

$$\min_{\theta_c} \mathcal{L}_C(\theta_c; \theta_s, \phi), \quad \max_{\theta_s} \mathcal{R}_S(\theta_s; \theta_c, \phi)$$θc​min​LC​(θc​;θs​,ϕ),θs​max​RS​(θs​;θc​,ϕ)

The Solver objective is determined directly by the Judge scores, whereas the Challenger objective incorporates auxiliary constraints to discourage repetitive or malformed tasks. The Challenger loss is defined as the negative expected composite reward.

To compute the Challenger reward, the authors first calculate the mean Judge score over the Solver responses for each task, which estimates how well the current Solver handles the task. The task difficulty reward is defined as one minus this mean score. To prevent the Challenger from generating near-duplicate tasks or malformed outputs, the difficulty reward is augmented with a repetition penalty and a format check. The repetition penalty groups tasks with high pairwise similarity into clusters and penalizes tasks based on their cluster size. The composite Challenger reward combines these elements, assigning a severe penalty if the generated task fails the format check. The Challenger parameters are then optimized via Group Relative Policy Optimization to maximize this composite reward.

For the Solver evolution, the authors freeze the Challenger and sample a larger pool of candidate tasks. They retain the tasks that provide the most informative training signal by measuring the response-level score dispersion. Tasks with the largest score dispersion are selected, as they lie near the current capability frontier of the Solver and offer the greatest room for learning. The Solver is then trained on these selected tasks using Group Relative Policy Optimization, holding the Challenger and Judge fixed. Through these alternating updates, the Challenger continually expands the task frontier, while the Solver adapts to produce increasingly high-quality responses.

Judge Adaptation To enable sustained self-improvement, the Judge co-evolves with the other two components. The authors impose two requirements on this co-evolution: preference pairs must be constructed entirely within the closed loop without external supervision, and their labels must not depend on signals produced by the Judge itself. They exploit two complementary sources of supervision to satisfy these requirements.

The first source is role-asymmetry pairs. For each held-out task, the chosen response is sampled from the Solver, while the rejected response is produced by asking the Challenger to solve its own task under the same prompt. Because the Solver is optimized to answer tasks well and the Challenger is optimized to make tasks difficult without receiving a learning signal for answering them, the Solver responses are systematically superior. This ordering is induced by the policy roles rather than the current Judge scores, providing discriminative supervision even when the Judge is miscalibrated.

The second source is subtask-amplification pairs. Relying solely on role-asymmetry pairs might cause the Judge to saturate at the current Solver capability level. To construct responses above this frontier, the authors adopt iterated amplification. The Challenger decomposes a held-out task into easier subtasks. The Solver answers each subtask, and the Challenger composes the partial solutions into a complete response. This amplified response is compared against a one-shot response sampled from the Solver. Because the Solver is more reliable on easier subtasks, the composed response tends to be more accurate, exposing the Judge to response quality above the current one-shot frontier.

The union of these two preference-pair sets forms the training data for the Judge. Starting from the parameters obtained in the previous iteration, the Judge is updated by minimizing the Bradley-Terry loss:

$$\mathcal{L}_J(\phi) = - \mathbb{E}_{(x, y^+, y^-) \sim \mathcal{D}} \left[ \log \sigma \left( J_\phi(x, y^+) - J_\phi(x, y^-) \right) \right]$$LJ​(ϕ)=−E(x,y+,y−)∼D​[logσ(Jϕ​(x,y+)−Jϕ​(x,y−))]

By focusing on the current frontier of self-evolution, the Judge learns to correct its misrankings on challenging examples, acquiring evaluation capability tailored to the latest policies it supervises.

Experiment

The experiments evaluate J-ZERO against base models and two zero-data self-play frameworks, R-Zero and G-Zero, on both verifiable and unverifiable benchmarks using Qwen3-4B-Base and Qwen3-8B-Base. J-ZERO achieves the best scores across all benchmark groups, with particularly large gains in unverifiable domains where baselines struggle due to their reliance on verifiable-only reward signals. Analysis shows that the self-generated preference labels remain reliable throughout training, with role-asymmetry pairs providing early signal and subtask-amplification pairs taking over once the solver matures, while ablations confirm both pair types contribute to performance. Crucially, judge co-evolution prevents the plateau observed in fixed-judge variants and competing methods, enabling sustained improvement over ten iterations as the judge's evaluation standard rises alongside the solver's capabilities.

Across verifiable mathematical reasoning benchmarks, J-ZERO achieves the highest average performance at both model scales, improving clearly over the base models and prior zero-data self-evolution methods. Gains are strongest on more difficult benchmarks such as Minerva, OlympiadBench, and AMC23. On GSM8K, scores are already high and R-Zero remains slightly ahead of J-ZERO at both scales. J-ZERO attains the best overall verifiable benchmark average among the compared methods at both 4B and 8B scales. Harder mathematical reasoning tasks such as Minerva, OlympiadBench, and AMC23 show clear improvements for J-ZERO, while GSM8K remains close and R-Zero is slightly ahead there.

On unverifiable benchmarks, J-ZERO consistently outperforms the base models and prior zero-data baselines at both evaluated scales. R-Zero and G-Zero show much smaller gains than in verifiable settings, leaving them closer to the base model, while J-ZERO's largest improvement occurs on the broad instruction-following AlpacaEval benchmark. J-ZERO leads on every unverifiable benchmark and overall average for both model scales, with gains substantially larger than those of R-Zero and G-Zero. The baselines improve only modestly: R-Zero is limited by a majority-vote reward that does not extend to open-ended tasks, and G-Zero without a Judge remains barely above the base model. The strongest J-ZERO gains appear on AlpacaEval, where it moves from low base-model scores to the highest reported scores at both scales.

Ablation results show that both Judge training data components contribute to performance, with subtask amplification providing the larger individual benefit. Removing subtask-amplification pairs reduces overall performance more than removing role-asymmetry pairs. Combining both sources yields the strongest results in verifiable, unverifiable, and overall metrics. The full J-ZERO configuration outperforms the fixed-Judge variant and the variants missing either preference data component. Subtask-amplification pairs are the stronger single contributor, as their removal causes a larger overall drop than removing role-asymmetry pairs. The two data sources are complementary and are most reliable at different stages, so using both together gives the best overall performance.

J-ZERO outperforms baseline and prior zero-data methods on verifiable math benchmarks, with the largest gains on harder tasks while GSM8K remains close to R-Zero. On unverifiable benchmarks, J-ZERO leads across all tasks and scales, showing especially strong improvement on AlpacaEval where prior approaches remain near the base model. Ablations indicate that both Judge training data sources contribute, with subtask amplification providing the larger individual benefit and the combination yielding the best overall results.