One-sentence Summary

Researchers from Johns Hopkins University, UC San Diego, the University of Exeter, and an independent researcher propose CO-RL, a cooperative multi-agent framework in which parameter-decoupled models are optimized via peer-derived rewards, with cohort diversity across model families, sizes, and rephrased samples reducing correlated errors; without ground-truth labels it yields gains of 3.0–8.6% across seven text-only benchmarks and 2.3–7.2% across four multimodal benchmarks.

Key Contributions

• The paper introduces CO-RL, a label-free multi-agent reinforcement learning framework in which multiple decoupled models, sharing no parameters, are optimized simultaneously using rewards derived from peer predictions rather than ground-truth labels or external judges.
• The paper shows that increasing cohort diversity through heterogeneous model families, sizes, and rephrased training samples reduces correlated errors and self-reinforcing feedback loops, improving reasoning performance, preserving behavioral diversity, and mitigating training collapse; theoretical analysis indicates that cross-agent supervision expands the set of initial conditions that converge to correct solutions.
• Across text-only and multimodal domains, CO-RL yields average gains of 3.0 to 8.6 percent across seven text-only LLM benchmarks and 2.3 to 7.2 percent across four multimodal VLM benchmarks, outperforming base models and prior label-free approaches and matching or surpassing supervised methods without ground-truth labels.

Introduction

Reinforcement learning with verifiable rewards has improved reasoning in large language models, but it depends heavily on ground-truth labels that become costly and scarce as tasks grow harder. Prior label-free approaches often rely on a single model’s own outputs for self-rewarding, which can amplify existing biases, reduce response diversity, and lead to training collapse. The authors introduce CO-RL, a cooperative multi-agent framework in which independently trained and diverse models supervise one another using peer majority-vote pseudo-answers. This decorrelated cross-agent supervision avoids external judges and ground-truth labels, improves text-only and multimodal reasoning, and often matches or surpasses supervised training.

Method

Reinforcement learning (RL) has become a standard approach for enhancing the reasoning capabilities of autoregressive language models. Typically, a policy $\pi_\theta$πθ​ generates a response $y$y for a prompt $x$x, and is optimized using a scalar reward $r(x, y)$r(x,y) via algorithms like Group Relative Policy Optimization (GRPO). In the absence of external verifiers, self-rewarding RL methods construct reward functions directly from the model's own outputs, such as using majority-vote agreement. However, because these signals originate from the same policy being optimized, training can reinforce existing biases, reduce response diversity, and eventually lead to homogenized responses or training collapse.

As shown in the figure below:

To address this, the authors introduce Co-Reinforcement Learning (Co-RL), a label-free multiagent RL framework. The core insight is that an independent learning signal can emerge from independently trained models, as their errors are decorrelated. One model can provide corrective feedback that another cannot derive from its own generations.

Refer to the framework diagram:

In Co-RL, a cohort of $N$N agents with independently parameterized policies $\{\pi_{\theta_n}\}_{n=1}^N${πθn​​}n=1N​ generates completions for unlabeled problems. The agents share neither parameters nor gradients; their optimization is coupled solely through the rewards they provide to one another. For each unlabeled problem $x$x, each agent $n$n independently rolls out a group of $K$K completions and extracts the final answers.

As shown in the figure below:

Crucially, each agent constructs a supervision target exclusively from the answers generated by a designated peer. The pseudo-label $\hat{a}_{-n}(x)$a^−n​(x) is constructed via majority vote over the peer's answers:

$$\hat{a}_{-n}(x) \in \arg\max_b \sum_{j=1}^K \mathbf{1}[a_{n-1}^j = b]$$a^−n​(x)∈argbmax​j=1∑K​1[an−1j​=b]

where the index is taken cyclically so that agent 1 is supervised by agent $N$N. The reward for the $k$k-th response of agent $n$n is then $r_n^k = \mathbf{1}[a_n^k = \hat{a}_{-n}(x)]$rnk​=1[ank​=a^−n​(x)]. This ensures that each agent does not contribute to its own supervision target.

Policy optimization is performed using GRPO. The rewards $\{r_n^k\}_{k=1}^K${rnk​}k=1K​ are normalized within each agent's rollout group to obtain group-relative advantages $\hat{A}_n^k$A^nk​. The overall training objective maximizes the average GRPO objective across all agents:

$$\max_{\boldsymbol{\theta}} \mathcal{J}_{\text{Co-RL}}(\boldsymbol{\theta}) = \frac{1}{N} \sum_{n=1}^N \mathcal{J}_{\text{GRPO}}\left(\theta_n; \{y_n^k, r_n^k\}_{k=1}^K\right)$$θmax​JCo-RL​(θ)=N1​n=1∑N​JGRPO​(θn​;{ynk​,rnk​}k=1K​)

where the per-agent objective includes the clipped surrogate loss and a KL regularization term against a reference policy.

The effectiveness of Co-RL relies heavily on cohort diversity. Highly similar models tend to make correlated errors, which weakens the corrective signal. The authors push diversity across three dimensions:

1. Decoupled policy optimization: Agents are trained independently with no gradient propagation between them, maintaining less-correlated predictions throughout the training process.
2. Model families and sizes: Using distinct pretrained model families (which differ in architecture, tokenization, and pretraining data) and different model sizes introduces orthogonal inductive biases and error profiles.
3. Input formation: Agents can be trained on semantically equivalent but differently phrased prompts (e.g., via rewriting) to reduce correlated errors induced by prompt-specific phrasing.

This diverse, decoupled setup allows Co-RL to exploit complementary strengths across agents, correcting errors that self-rewarding methods would otherwise reinforce.

Experiment

The experiments evaluate CO-RL on paired language and vision-language models trained on math-centric datasets, comparing against self-rewarding, multi-agent RL, and ground-truth supervised baselines across reasoning and multimodal math benchmarks. The main results show that CO-RL consistently improves over self-rewarding and multi-agent methods, scales to three heterogeneous agents, and transfers to vision-language models, often matching or exceeding ground-truth supervision without using labels. Ablations further confirm stable training dynamics without reward collapse or length degeneration, and budget-matched comparisons indicate that the gains stem from cross-agent supervision rather than additional compute or ensembling.

CO-RL outperforms label-free self-rewarding baselines on language benchmarks. The same-family setting already improves average performance over the Qwen2.5-3B and Llama-3.2-3B-Instruct base models by 8.0 and 4.0 percentage points, respectively, while cross-family diversity provides further gains overall. Ground-truth reward training serves only as a supervised reference. The same-family CO-RL variant improves average performance by 8.0 percentage points for Qwen2.5-3B and 4.0 percentage points for Llama-3.2-3B-Instruct over their base models. CO-RL leads the label-free methods, outperforming TTRL, RENT, Intuitor, and Co-Rewarding-II, with cross-family diversity adding further overall benefits.

In the CoMAS multi-agent RL comparison, the different-family CO-RL variant achieves the highest average performance and leads on five of seven benchmarks. It outperforms CoMAS by four percentage points while using half as many agents and no additional judging mechanism, and it also exceeds MAPoRL and TTRL on average. CO-RL with different-family agents attains the highest overall average among all compared methods. It leads on five of seven benchmarks and outperforms CoMAS by four percentage points while using half as many agents.

In a single three-agent CO-RL run with heterogeneous model families, CO-RL improves all three base models, with average gains ranging from about six to eight percentage points. Without using ground-truth supervision, three-agent CO-RL matches or exceeds ground-truth reward training on average across the models. It also outperforms the TTRL self-rewarding baseline for Qwen2.5-3B and Llama-3.2-3B-Instruct. CO-RL consistently improves over base performance for all three heterogeneous agents. Three-agent CO-RL matches or surpasses GT-Reward on average without ground-truth supervision. CO-RL outperforms the TTRL self-rewarding baseline for Qwen2.5-3B and Llama-3.2-3B-Instruct.

For the small vision-language setting, CO-RL improves over the base model and TTRL for InternVL3.5-2B under open-r1, achieving the highest average among the compared methods and slightly surpassing ground-truth supervision. The reported results also note that CO-RL attains the best average in three of four small 2B-3B settings while remaining competitive with ground-truth rewards. For MMR1, TTRL shows clear gains over the base model and posts a higher average than ground-truth supervision. Under open-r1, CO-RL edges out both TTRL and GT-Reward on average for InternVL3.5-2B. In per-benchmark open-r1 results, GT-Reward still leads on MathVision, MathVerse, and MathVista, while TTRL leads on the paper-Math. For MMR1, TTRL improves average accuracy above the base model and slightly above GT-Reward.

The experiments evaluate CO-RL as a label-free self-rewarding method across language benchmarks, multi-agent reinforcement learning, heterogeneous three-agent runs, and small vision-language models. CO-RL consistently improves over its base models and outperforms label-free baselines such as TTRL, RENT, Intuitor, and Co-Rewarding-II, with cross-family diversity providing additional gains. In multi-agent comparisons, the different-family variant achieves the highest overall average while using fewer agents and no separate judging mechanism, and a heterogeneous three-agent run improves all participating models while matching or exceeding ground-truth reward training without supervision. In the small vision-language setting, CO-RL leads for InternVL3.5-2B and remains competitive with ground-truth supervision, while TTRL shows stronger gains for MMR1.