HyperAIHyperAI

Command Palette

Search for a command to run...

MARCH : Vérification multi-agents renforcée contre les hallucinations

Résumé

L'hallucination demeure un goulot d'étranglement critique pour les grands modèles de langage (LLM), compromettant leur fiabilité dans les applications du monde réel, en particulier dans les systèmes de génération augmentée par récupération (RAG). Alors que les méthodes existantes de détection des hallucinations utilisent des LLM en tant que juges pour vérifier les sorties des LLM par rapport aux preuves récupérées, elles souffrent d'un biais de confirmation inhérent, où le vérificateur reproduit involontairement les erreurs de la génération d'origine. Pour remédier à cela, nous introduisons MARCH (Multi-Agent Reinforced self-Check for Hallucination), un cadre qui impose un alignement factuel rigoureux en exploitant une asymétrie d'information délibérée. MARCH orchestre un pipeline collaboratif de trois agents spécialisés : un solveur, un proposeur et un vérificateur. Le solveur génère une réponse RAG initiale, que le proposeur décompose en propositions atomiques vérifiables au niveau des affirmations. De manière cruciale, le vérificateur valide ces propositions par rapport aux preuves récupérées de manière isolée, sans avoir accès à la sortie originale du solveur. Ce schéma d'asymétrie d'information bien conçu rompt le cycle du biais d'auto-confirmation. En entraînant ce pipeline avec l'apprentissage par renforcement multi-agents (MARL), nous permettons aux agents de co-évoluer et d'optimiser l'adhésion factuelle. Des expériences approfondies sur des références d'hallucination démontrent que MARCH réduit considérablement les taux d'hallucination. Notamment, un LLM de 8 milliards de paramètres équipé de MARCH atteint des performances comparables à celles de modèles propriétaires puissants. MARCH ouvre une voie évolutive pour l'auto-amélioration factuelle des LLM grâce à la co-évolution. Le code est disponible sur https://github.com/Qwen-Applications/MARCH.

One-sentence Summary

Researchers from Alibaba and The Chinese University of Hong Kong, Shenzhen, propose MARCH, a multi-agent reinforcement learning framework that reduces LLM hallucination in RAG by orchestrating Solver, Proposer, and Checker agents with deliberate information asymmetry to break confirmation bias, enabling an 8B8\text{B}8B-parameter model to achieve performance competitive with closed-source systems.

Key Contributions

  • MARCH (Multi-Agent Reinforced self-Check for Hallucination) orchestrates three specialized agents: a Solver generating the RAG response, a Proposer decomposing it into atomic claim-level propositions, and a Checker validating these claims against retrieved evidence in isolation, deliberately blinded to the Solver's original output. This information asymmetry breaks the cycle of self-confirmation bias that plagues conventional LLM-as-a-judge verification.
  • The framework trains the pipeline end-to-end with multiagent reinforcement learning (MARL) using a zero-tolerance reward mechanism, where any discrepancy between extracted claims and document-based validation triggers a penalty on the entire response trajectory. This joint policy optimization enables the agents to co-evolve and internalize a self-contained verification loop, distinct from prior multi-agent systems that treat agents as separate entities without shared policy training.
  • Experiments across hallucination benchmarks show that MARCH substantially reduces hallucination rates without requiring additional human annotations or external fact-checking tools. Notably, an 8B-parameter LLM equipped with MARCH achieves performance parity with leading closed-source models on multiple hallucination benchmarks while also delivering significant gains in general RAG-QA tasks, and the framework generalizes across different model families.

Introduction

Large language models still struggle with factuality even when augmented with retrieval, producing fluent answers that contradict the provided source material. This problem is especially severe in data-intensive domains like finance, law, and healthcare, where users expect models to act as rigorous evidence readers rather than plausible storytellers. The authors note that prior mitigation efforts fall short in several ways. Supervised fine-tuning can actually amplify hallucinations by promoting stylistic imitation over factual fidelity, while the scalar rewards used in RLHF are too coarse-grained to supervise claim-level accuracy. Verifiable-reward approaches depend on scarce expert-annotated ground truth and are bounded by the reasoning limits of external verifiers. Perhaps most critically, existing evaluation schemes expose verifiers to the query, documents, and generated response simultaneously, inducing a confirmation bias that rewards internal coherence rather than genuine source grounding. To address this, the authors introduce MARCH, a reinforcement learning framework built on deliberate information asymmetry. It orchestrates three specialized agents: a Solver generates responses, a Proposer atomizes them into discrete verifiable claims, and a Checker validates those claims against the documents while strictly blinded to the Solver's output, preventing cognitive contamination. Training employs a zero-tolerance reward mechanism that penalizes any mismatch between extracted claims and document-based validation, pushing the system toward self-verification without external fact-checkers or extra annotations. Notably, an 8B-parameter model equipped with MARCH matches leading closed-source systems across several hallucination benchmarks while also improving general RAG question answering, demonstrating a scalable path to trustworthy agentic self-improvement.

Dataset

The authors construct a training setup using three publicly available question answering datasets, split into two distinct settings based on domain and difficulty.

  • STEM setting: Built on BioASQ, focusing on science, technology, engineering, and math queries.
  • General setting: Built on 2WikiMultiHopQA and MuSiQue, covering more complex multi-hop reasoning tasks.

For all datasets, the training data consists only of the original queries and a set of retrieved documents, with no access to ground-truth answers or annotations. This makes the setup realistic and challenging, because the retrieval process introduces substantial noise and heterogeneity.

Key statistics per subset (from Table 1):

  • The number of retrieved documents per query and document lengths vary widely across datasets.
  • A large fraction of retrieved documents are irrelevant to the query:
    • BioASQ (STEM): 30.1% irrelevant
    • MuSiQue (General): up to 88.4% irrelevant
    • 2WikiMultiHopQA (General): also shows high noise, though not explicitly quantified in the text

This high level of noise forces the agents to learn document-grounded fact checking and reasoning in the presence of many distractors.

The authors use these datasets in their training framework, starting from the Meta-Llama3.1-8B-Instruct base model. The training is performed via a reinforcement learning pipeline (based on VerL), but the dataset description itself covers only the composition, sources, and noise characteristics. No explicit filtering or preprocessing steps are mentioned; the retrieved documents are used as-is, preserving their original noisy distribution.

Method

The authors propose MARCH (Multi-Agent Reinforced self-Check for Hallucination), a framework that reformulates factual grounding as a self-contained, verifiable objective by decomposing the generation process into a structured, asymmetric cooperative game. Instead of relying on unavailable oracle ground truth, the method instantiates three specialized agents derived from a single base policy πθ\pi_\thetaπθ​: a Solver, a Proposer, and a Checker.

The pipeline begins with the Solver agent, which generates a comprehensive response yyy conditioned on the input query xxx and retrieved documents DDD. To enable granular verification, the Proposer agent functions as a response atomizer, decomposing the continuous narrative yyy into a set of discrete, self-contained factual question-answer pairs {(qi,ai)}i=1n\{(q_i, a_i)\}_{i=1}^n{(qi​,ai​)}i=1n​. Finally, the Checker agent acts as a blinded auditor. It answers the questions {qi}\{q_i\}{qi​} generated by the Proposer using only the retrieved documents DDD, strictly without access to the original response yyy or the extracted claims aia_iai​. This information asymmetry eliminates confirmation bias. To ensure a reliable baseline for auditing, the Checker employs a multi-sampling strategy where multiple independent audit trajectories are generated for each question, followed by majority voting to determine the final consensus answer a^i\hat{a}_ia^i​.

As shown in the figure below:

The verification loop utilizes a Zero-Tolerance Reward (ZTR) mechanism to enforce absolute factual grounding. This binary reward function assigns a score of 0 only if every claim aia_iai​ asserted by the Solver achieves an exact identity match with the independent grounding a^i\hat{a}_ia^i​ provided by the Checker. Any hallucination results in a score of -1, creating an "all-or-nothing" structure that prevents the policy from optimizing for partial correctness.

The shared policy πθ\pi_\thetaπθ​ is optimized using Proximal Policy Optimization (PPO). The training process incorporates both the reasoning path yyy from the Solver and the audit trajectory λ\lambdaλ from the Checker into the optimization loop. The joint gradient is computed by summing the advantages estimated from both trajectories, allowing the model to concurrently refine its narrative generation and evidentiary precision while maintaining a KL-divergence penalty against the reference model.

Experiment

The experiments evaluate MARCH on hallucination mitigation and downstream QA reasoning. Trained solely on noisy, document-grounded RAG data without answer annotations, MARCH consistently improves factual consistency and nuanced judge behavior across diverse hallucination benchmarks, elevating an 8B model to a level comparable to much larger proprietary systems. The improved grounding directly translates into better multi-hop reasoning performance, with the small model surpassing larger baselines on noisy benchmarks. Ablations confirm that jointly optimizing the Solver and Checker with a zero-tolerance reward is most effective, and the framework generalizes across model families, though the Proposer tends to prune risky claims over time, which can be mitigated with simple instructional constraints.

The training datasets show substantial heterogeneity, with large variance in the number of retrieved documents and their lengths, as well as a high proportion of irrelevant documents across all settings. This noise and variability make the training environment realistic but challenging for learning document-grounded fact checking without ground truth annotations. The ratio of irrelevant documents per query varies widely across datasets, reaching more than 88% in one setting. Retrieved document counts and document lengths exhibit extreme ranges, from single documents to tens of thousands of words. The datasets cover both STEM and general domains, sourced from different original datasets, contributing to the diversity in query and document statistics.

该表比较了多种模型在RAGTruth和FaithBench基准上的一致性率,结果显示不同模型的表现差异显著,其中GLM-4-9B-Chat的平均一致性率最高,而Phi-4-mini-Instruct在RAGTruth上表现尚可但在FaithBench上明显较弱。 GLM-4-9B-Chat达到了最高的平均一致性率(73.06%),优于其他对比模型。 Phi-4-mini-Instruct在RAGTruth上得分较高(78.67%)但FaithBench上仅24.67%,显示出跨基准的显著不一致性。 多数模型在RAGTruth上的得分高于FaithBench,表明不同基准对事实一致性的评估难度存在差异。

The table compares several large judge models on ContextualJudgeBench across refusal, faithfulness, completeness, and conciseness metrics for QA and summarization tasks. SFRJudge-70B achieves the highest average accuracy, while Promtheus-2-8x7b performs substantially worse across all dimensions. Most models show notable variability across subtasks, with refusal detection on unanswered questions being particularly challenging. SFRJudge-70B leads the average accuracy at 51.4%, outpacing the next best model by about 9.5 points. Promtheus-2-8x7b has the lowest overall accuracy, especially on faithfulness and completeness for both QA and summarization. Models often score much lower on refusal for unanswered questions, such as Skywork-70B at 11.2%, compared to their refusal scores for answered questions.

MARCH-Best, an 8B-class model, outperforms specialized RAG baselines and even exceeds several GPT-4o based frameworks on multi-hop QA benchmarks. Notably, it achieves higher accuracy on HotpotQA and MuSiQue compared to methods like RAFT and GenGround, and its HotpotQA result surpasses that of standard GPT-4o RAG and IRCoT with GPT-4o. MARCH-Best achieves 71.2% on HotpotQA, exceeding standard GPT-4o RAG (64.0%) and IRCoT with GPT-4o (66.4%). Among 8B-class models, MARCH-Best shows higher accuracy on HotpotQA and MuSiQue than baselines such as RAFT and GenGround. The framework improves factual grounding in smaller models, yielding performance comparable to or better than much larger proprietary models on multi-hop reasoning tasks.

The ablation study compares reward mechanisms and configurations for the MARCH framework. Results show that the zero-tolerance reward (ZTR) outperforms the error-rate reward (ERR), and penalty-based scalar settings are more effective than incentive-based ones. Joint optimization of the Solver and Checker yields the highest accuracy, and the framework generalizes across different model families. Zero-tolerance reward consistently surpasses error-rate reward, indicating strict logical consistency benefits complex RAG tasks. Within ZTR, the penalty-based scalar (-1/0) outperforms the incentive-based (0/1) setting, likely due to a stronger corrective gradient against hallucinations. On Qwen3-8B, the framework achieves over 11% absolute improvement over its vanilla baseline, demonstrating model-agnostic generalization.

The experiments evaluate the proposed MARCH framework across multiple benchmarks, including RAGTruth, FaithBench, ContextualJudgeBench, and multi-hop QA datasets such as HotpotQA and MuSiQue. MARCH-Best, an 8B-class model, outperforms specialized RAG baselines and several GPT-4o based frameworks on multi-hop QA, while ablations show that a zero-tolerance reward with penalty-based scalars and joint Solver-Checker optimization yield the highest accuracy, indicating strict logical consistency is beneficial. The framework also generalizes across model families, achieving over 11% absolute improvement over a vanilla baseline on Qwen3-8B, despite the training data being noisy and heterogeneous.


Créer de l'IA avec l'IA

De l'idée au lancement — accélérez votre développement IA avec le co-codage IA gratuit, un environnement prêt à l'emploi et le meilleur prix pour les GPU.

Codage assisté par IA
GPU prêts à l’emploi
Tarifs les plus avantageux

HyperAI Newsletters

Abonnez-vous à nos dernières mises à jour
Nous vous enverrons les dernières mises à jour de la semaine dans votre boîte de réception à neuf heures chaque lundi matin
Propulsé par MailChimp