One-sentence Summary

Duke University, Adobe Inc., and others propose Reinforcement Learning with Self-Verifiable Rewards (RLSVR), a task-transformation paradigm that converts open-ended tasks into verifiable proxy environments, instantiated as SpyRL, a self-play game based on 'Who Is the Spy?' whose predetermined spy identities yield fully verifiable reward signals, and demonstrate superior self-improvement over existing methods on text summarization, creative writing, and mathematical reasoning.

Key Contributions

• RLSVR extends RLVR to open-ended tasks by transforming them into proxy environments whose internal rules produce self-verifiable rewards, inspired by self-supervised pretext task construction.
• SpyRL instantiates RLSVR as an information-asymmetric multi-agent social deduction game, where predetermined spy identities and voting outcomes automatically generate verifiable rewards tied to output quality.
• Experiments on text summarization, creative writing, and mathematical reasoning show that SpyRL achieves 75.4% and 77.3% win rates on Qwen3-8B for summarization and creative writing, and improves mathematical reasoning by 8.97% and 6.16% on Qwen3-4B and 8B across seven benchmarks. Vote-based rewards align closely with human and LLM quality judgments.

Introduction

Reinforcement learning with verifiable rewards (RLVR) has successfully scaled the training of reasoning models for deterministic domains like math and coding, but it breaks down in open-ended tasks such as summarization or creative writing, where no ground-truth verifier exists. Prior approaches relax this requirement by using learned preference signals or model-based judges, but these introduce evaluation bias, bottlenecks tied to judge capability, and additional inference overhead. The authors propose Reinforcement Learning with Self-Verifiable Rewards (RLSVR), a paradigm that, like self-supervised learning, transforms the original task into a proxy environment where rewards become automatically verifiable. They instantiate this idea with SpyRL, an information-asymmetric self-play framework modeled after a social deduction game. In SpyRL, agents receive different amounts of task information, produce outputs, and then vote to identify which agent is the spy, a process that converts output quality assessment into a verifiable identity-inference problem. Experiments show that SpyRL substantially outperforms existing self-improvement methods on non-verifiable tasks and also yields consistent gains on verifiable mathematical reasoning.

Method

The authors propose RLSVR (Reinforcement Learning with Self-Verifiable Rewards) to address the verifiability bottleneck in RLVR for open-ended tasks. Instead of approximating an unverifiable objective with a learned reward model, RLSVR transforms the original task into a proxy environment where rewards are deterministic and rule-based. This transformation involves latent-variable injection, conditioned task execution, verifiable interaction, and rule-based reward computation, effectively turning self-supervised learning principles into a reinforcement learning context.

To instantiate this concept, the authors introduce SpyRL, a multi-agent self-play framework that transforms the unverifiable objective of output quality into a verifiable identity-recognition problem.

As illustrated in the framework diagram, each SpyRL training epoch alternates between a Performing stage and a Detection stage. In the Performing stage, $n-1$n−1 civilian players receive full information while one spy player receives a degraded version of the input. All players generate outputs for the target task. In the Detection stage, players jointly analyze these outputs to identify the spy. Because the environment explicitly assigns the spy identity, the detection yields a naturally verifiable outcome. The rewards across both stages are tightly coupled: the preassigned identity supervises the detection stage, whose outcomes simultaneously determine the performing stage's reward.

In the Information-Asymmetric Performing Stage, the authors design capability-oriented tasks to ensure the required skills align with the target task. To introduce a verifiable training signal, SpyRL assigns asymmetric information inputs. For each player $i$i, the private observation $o_i$oi​ is constructed such that civilians receive the original instance $x$x, while the spy receives a degraded version $g(x)$g(x), where $g(\cdot)$g(⋅) is an information-degradation operator (e.g., context truncation or key information compression).

The figure above details this mechanism across three representative domains: text summarization, creative writing, and mathematical reasoning. The degradation operator obscures critical information while preserving style and thematic consistency, ensuring the spy exhibits inferior performance and establishing a verifiable reward signal. Each player generates an output $y_i \sim \pi_\theta^P(\cdot \mid o_i, \tau)$yi​∼πθP​(⋅∣oi​,τ), incentivized to perform at their highest capacity to avoid being voted out as the spy.

In the Detection Stage with Verifiable Rewards, each player infers the spy's identity based on the public outputs $Y = \{y_1, \dots, y_n\}$Y={y1​,…,yn​}. Player $i$i samples a voting action $v_i \sim \pi_\phi^D(\cdot \mid s_i)$vi​∼πϕD​(⋅∣si​), where $s_i = (o_i, Y)$si​=(oi​,Y). Since the spy identity $u$u is explicitly specified by the environment, the base reward for detector $i$i is $r_i^D = \mathbb{I}[v_i = u]$riD​=I[vi​=u]. To obtain an optimization signal with lower variance, the authors adopt a GRPO-style group relative advantage formulation, normalizing the rewards within the group to allow detectors to learn without an additional critic network.

The training of SpyRL relies on Two-Stage Coupled Optimization. The voting results from the detection stage define the reward for the performing stage, creating a closed-loop learning system. The reward design for performers follows a zero-sum constraint between the spy and civilian players. The spy player $u$u receives a reward $r_u^P = -\beta(m_u - \bar{m}_c)$ruP​=−β(mu​−mˉc​), while civilian players $c_j$cj​ receive $r_{c_j}^P = \frac{\beta}{n_c}(m_u - \bar{m}_c) - \lambda(m_{c_j} - \bar{m}_c)$rcj​P​=nc​β​(mu​−mˉc​)−λ(mcj​​−mˉc​), where $m$m denotes the number of votes received. This ensures continual co-evolution through competition. To alleviate systemic bias induced by information asymmetry, Role-Advantage Estimation (RAE) is incorporated during optimization.

During iterative optimization, the performer policy $\pi_\theta^P$πθP​ and detector policy $\pi_\phi^D$πϕD​ are updated separately using a GRPO-style clipped objective with KL regularization. The authors alternate between optimizing the performing stage and the detection stage to avoid premature convergence. When detectors easily identify the spy, optimization shifts to performers to generate higher-quality, more deceptive outputs. Conversely, when identification becomes too difficult, training switches back to the detection stage. This alternating strategy breaks policy stagnation and reduces interference in credit assignment, leading to better training stability and sample efficiency.

Experiment

The evaluation uses a self-play game across three domains where a spy receives masked input and civilians receive the full text, with alternating training of performing and detection stages. SpyRL reformulates quality improvement as an identity-discrimination problem, aligning rewards with task performance without external verifiers. It consistently outperforms baselines on both verifiable and non-verifiable tasks, with gains confirmed by human evaluation and broad generalization, while ablations highlight the necessity of alternating optimization, the spy mechanism, and role-advantage calibration.

On summarization benchmarks, SpyRL consistently achieves the highest ROUGE-L scores, outperforming the base model and the R-Zero and Absolute Zero baselines for both Qwen3-4B and Qwen3-8B backbones. In GPT-4o A/B evaluations, SpyRL wins the majority of pairwise comparisons against every other method in all thirty tested settings, confirming its superior generation quality. SpyRL attains the highest ROUGE-L on all five summarization datasets regardless of backbone, with margins over the base model and baselines that are particularly large on GovReport. In GPT-4o pairwise A/B tests, SpyRL's win rate exceeds 50% against every other method (base, R-Zero, Absolute Zero) on every benchmark and for both model sizes.

On creative writing benchmarks, SpyRL outperforms its untrained backbone and the self-evolution baselines in every fine-grained dimension for both Qwen3-4B and Qwen3-8B backbones. The largest pairwise win margins appear in novelty and emotion, showing that improvements extend beyond surface fluency to more subjective aspects of open-ended generation. SpyRL wins all pairwise comparisons against the backbone and both baselines across all dimensions and backbones, with overall win rates ranging from 71.1% to 81.3%. The largest gains are in novelty and emotion, where SpyRL's win rate against the Qwen3-4B backbone reaches 84.3% on WritingPrompt novelty, while coherence and consistency scores are comparatively lower, indicating deeper improvements in subjective quality.

SpyRL achieves the highest accuracy on all seven mathematical and general reasoning benchmarks for both Qwen3-4B and Qwen3-8B, outperforming the backbone and baseline methods. Gains are especially large on the most challenging math benchmarks, and the method also lifts performance on knowledge-intensive reasoning tasks like MMLU-Pro and GPQA-D, indicating its benefits extend beyond pure mathematics. SpyRL attains the best accuracy on every benchmark for both model sizes, surpassing the backbone, R-Zero, and Absolute Zero. On AIME25 with Qwen3-4B, SpyRL improves accuracy from 6.7% to 20.0%, more than doubling the backbone performance. The method boosts MMLU-Pro and GPQA-D scores, demonstrating gains on broader reasoning tasks beyond math. SpyRL's advantage is consistent across all five math benchmarks, with the largest relative improvements on the difficult AIME24 and AIME25 sets.

In human evaluations of creative writing, SpyRL is preferred over the backbone model, R-Zero, and Absolute Zero on every dimension and dataset. The strongest wins occur in novelty and emotion, showing that SpyRL's gains extend beyond fluency to more subjective qualities of open-ended text. SpyRL wins the majority of pairwise comparisons across all five dimensions on both WritingPrompt and WritingBench, with overall win rates ranging from 72.0% to 85.0%. The largest margins are in novelty and emotion, where win rates against the backbone reach 84.5% and 85.0% on WritingBench, indicating improved subjective richness. Against the strongest baseline, R-Zero, SpyRL still secures a clear advantage, exceeding 80% win rate in overall quality on both datasets.

SpyRL consistently wins against Qwen3.5-RaR across all dimensions on both writing benchmarks, with overall win rates of 59.3% for WritingPrompt and 56.2% for WritingBench. Against GPT-4o-RaR, win rates are close to 50%, indicating competitive performance. Crucially, SpyRL achieves these results without an external verifier, while the baselines incur $200 and$200and900 in additional costs. On WritingPrompt, SpyRL wins against Qwen3.5-RaR in every dimension, with an overall win rate of 59.3%. Against GPT-4o-RaR, SpyRL's overall win rates are 48.9% and 48.2%, showing comparable quality while avoiding the $200–$200–900 verifier cost.

SpyRL is evaluated across summarization, creative writing, reasoning, and human preference benchmarks, consistently outperforming its untrained backbone and self-evolution baselines such as R-Zero and Absolute Zero. The method delivers particularly large gains in subjective qualities like novelty and emotion on creative tasks, and achieves the highest accuracy on all reasoning benchmarks, with pronounced improvements on challenging math problems. Human evaluations confirm its preference over baselines, and it matches the quality of costly verifier-based approaches without incurring additional verifier costs, demonstrating robust, cost-effective self-improvement.