One-sentence Summary

Researchers from the University of Chinese Academy of Sciences, Tsinghua University, Peking University, Beijing Institute of Technology, East China Normal University, and University of Science and Technology of China propose Multi-Agent Protocol Distillation (MAPD), a joint distillation and reinforcement learning framework that distills structured JSON protocols from a multi-agent exploration trace into a privileged student branch to bridge the proprietary-to-open-source distribution gap for agentic search, achieving up to 44.4% success rate on Qwen3-4B and mitigating style drift.

Key Contributions

• Multi-Agent Protocol Distillation (MAPD) is a joint distillation and reinforcement learning framework that decouples a teacher model's reasoning strategy from its linguistic style by using a structured JSON protocol as an intermediate representation, thereby providing dense supervision without requiring logit access or matching tokenizers.
• An offline multi-agent system synthesizes this style-normalized protocol from proprietary teacher trajectories, capturing task type, reasoning plan, and extractive grounding facts, and serves as privileged information to guide a student policy's token distributions during training alongside a sparse RL objective.
• Experiments across seven knowledge-intensive QA benchmarks show that MAPD consistently outperforms competitive distillation and RL baselines, achieving average success rates of 39.4% on Qwen3-1.7B and 44.4% on Qwen3-4B, while generalizing robustly across different proprietary teachers and mitigating style drift and verbosity degeneration.

Introduction

Large language models are increasingly applied to multi-turn agentic tasks such as agentic search, where a policy must interleave reasoning with retrieval to answer complex, knowledge-intensive queries. Training these systems typically relies on outcome-based reinforcement learning, which provides only sparse supervision, or supplements it with online policy distillation that aligns student and teacher distributions at the token level. This distillation, however, requires a high-quality teacher, and while proprietary models offer strong reasoning capabilities, distilling from them introduces two core obstacles: mismatched tokenizers and inaccessible logits make token-level alignment infeasible, and naive trajectory imitation forces open-source students to mimic verbose, teacher-specific styles, causing distribution drift, hallucinations, and poor transfer of strategic reasoning.

The authors address this by introducing Multi-Agent Protocol Distillation (MAPD), a joint framework that combines on-policy self-distillation with outcome-based reinforcement learning. MAPD decouples cognitive strategies from linguistic surface forms by synthesizing a style-normalized JSON protocol from a multi-agent system powered by proprietary teachers, used only during offline training. During online alignment, a protocol-conditioned branch of the student model is aligned via dense self-distillation, complementing sparse reward signals without ever needing access to teacher logits or adding inference overhead.

Dataset

The authors construct a training dataset of structured JSON protocols through an automated three-stage multi-agent pipeline. The dataset serves as the target for training a student model, with ground-truth answers strictly hidden from the student.

Dataset composition and sources

• Each example is a structured JSON protocol derived from a single question.
• The pipeline uses a local Wikipedia corpus for retrieval; the original question source is not detailed but is assumed to be a QA dataset.
• Two protocol variants are produced:
  • Succeeded Protocol – includes a fully verified answer, a step-by-step reasoning plan, and extractive grounding facts.
  • Evidence Protocol – generated when the search fails; omits a definitive answer and instead provides a neutral summary of partial findings.
• All protocols are automatically generated by a Protocolizer agent that converts an exhaustive exploration log (sub-tasks, queries, retrieved passages, findings) into the JSON target.

Key details for each subset

• Succeeded Protocol subset: contains examples where the multi-agent search converged to the correct answer. The protocol includes the verified answer, a reasoning plan formulated without hindsight, and grounding facts that are verbatim substrings from retrieved passages.
• Evidence Protocol subset: contains examples where the search did not converge. The answer field is replaced with a neutral evidence summary (e.g., “Confirmed …, but reliable evidence for … is missing”). The reasoning plan and grounding facts follow the same strict constraints.
• The pipeline generates both types automatically, conditioned on a binary success label from Stage A.

Data processing and quality control

• All generated protocols pass through a quality gate with four automated checks:
  • Schema Validation (well-formed JSON)
  • EM Consistency (for succeeded protocols, answer matches ground truth)
  • Grounding Verification (facts are extractive substrings)
  • Leak Detection (no oracle knowledge in reasoning steps)
• Protocols failing any check are discarded; training falls back to a self-rollout baseline for those instances.
• Manual review of 3,000 instances across three models shows 99.33% accuracy in yielding compliant protocols.

How the paper uses the data

• The final dataset of high-quality JSON protocols is used as the training target for the student model.
• The authors do not specify an explicit train/validation split or mixture ratio, but both Succeeded and Evidence protocols are included to teach the model both answer generation and evidence summarization.
• The dataset is not used for evaluation; it is exclusively a training resource, with ground-truth answers never exposed to the student.

Method

The authors frame agentic search as a multi-turn interaction between a language model policy and a retrieval-augmented environment. The policy generates reasoning steps and optionally issues retrieval queries, receiving relevant passages as observations, until a final answer is emitted or a turn limit is reached. Each trajectory receives a sparse binary reward based on exact-match evaluation of the final answer. To optimize this policy, the method builds on Group Relative Policy Optimization (GRPO), which samples a group of rollouts per prompt, computes advantages from terminal rewards, and updates the policy using a clipped surrogate objective with a per-token KL penalty against a reference model. However, GRPO alone provides only outcome-level supervision, which is insufficient for refining complex intermediate reasoning.

To inject denser, token-level guidance, the authors adopt Online Policy Self-Distillation (OPSD). In OPSD, the current policy serves as both student and teacher: the student branch conditions on the standard context, while the teacher branch additionally receives privileged information (PI) that is available only during training. By minimizing the per-token reverse KL divergence between the two branches and restricting gradient flow to the student, the model distills knowledge from the richer teacher context without vocabulary mismatch issues. Standard OPSD, however, derives PI from the student's own correct rollouts, limiting supervision quality to the student's existing capabilities.

The proposed framework overcomes this limitation by introducing a Structured JSON Protocol and a Multi-Agent System (MAS) generation pipeline that produces high-quality privileged information from a proprietary teacher model, while decoupling cognitive strategies from teacher-specific linguistic patterns. The overall architecture is a joint training scheme that combines a token-level OPSD distillation signal with a sparse GRPO reinforcement learning signal.

The Structured JSON Protocol normalizes privileged information into a compact, traceable format containing five essential fields: task type (single_hop, multi_hop, comparison, others), an ordered reasoning plan of actionable sub-goals, grounding facts extracted verbatim from retrieved passages, optional partial findings for failed searches, and answer verification with a groundedness flag. By conditioning the teacher branch on this style-normalized protocol rather than free-form teacher text, the method reduces representation mismatch and mitigates style drift and hallucinations.

To automatically construct these protocols from a proprietary teacher, the authors design a three-stage MAS pipeline with rigorous quality control. In Stage A (Multi-Agent Collaborative Search), an Orchestrator agent decomposes the question into sub-tasks with dependency annotations, dispatches them to parallel Searcher agents that query a local Wikipedia corpus and compress results, and synthesizes findings across up to multiple rounds. If an exact-match check against the ground truth fails, a Repair agent uses the ground truth as diagnostic guidance (never exposed to the student) to identify expression or search issues and trigger targeted re-decomposition and additional searches. Stage B (Protocol Generation) employs a Protocolizer agent that converts the exhaustive exploration log into either a Succeeded Protocol (with verified answer) or an Evidence Protocol (with partial findings and no answer) based on the success label. Two integrity constraints are enforced: no hindsight in the reasoning plan, and extractive grounding of facts. Stage C (Quality Gate) applies four automated checks: schema validation, exact-match consistency for succeeded protocols, grounding verification via substring matching, and leak detection to prevent oracle knowledge injection. Protocols failing any check are discarded, and training falls back to a self-rollout baseline. Manual review confirms a 99.33% accuracy rate in yielding compliant protocols.

Finally, the joint training objective combines the OPSD loss and the GRPO loss:

where $\mathcal{L}_{\mathrm{OPSD}}$LOPSD​ provides token-level distillation from the structured protocol, enabling efficient step-by-step credit assignment, and $\mathcal{L}_{\mathrm{GRPO}}$LGRPO​ supplies a sparse RL signal via clipped policy gradients to optimize toward the final environment reward. The hyperparameter $\lambda_{\mathrm{OPSD}}$λOPSD​ controls the relative strength of the distillation guidance. This integrated approach allows the student model to benefit from dense, high-quality reasoning supervision while still being aligned to the downstream task through outcome-based reinforcement learning.

Experiment

The experiments train on NQ and HotpotQA and evaluate on seven single- and multi-hop QA benchmarks to assess MAPD against several pure and hybrid RL-distillation baselines, including GRPO, OPSD, and SDAR. MAPD consistently outperforms prior methods, with larger relative gains on complex multi-hop tasks that require decomposition and evidence aggregation. Ablations demonstrate that the structured protocol is essential to prevent style drift during cross-model distillation, while the multi-agent system enhances protocol quality and reasoning completeness. Additional analyses show the approach works robustly across different proprietary teacher backbones and identify an optimal distillation weight that balances retrieval efficiency with multi-step reasoning.

The MAPD framework consistently outperforms all baseline methods on both evaluated model sizes, with the largest gains appearing on multi-hop question answering benchmarks. Relative to the strongest baseline SDAR, MAPD provides average improvements of 4.8% and 3.3% on the 1.7B and 4B models, respectively. The advantage is especially pronounced for complex multi-hop tasks, where relative gains reach 7.9% and 5.1%, underscoring the benefit of multi-agent decomposition and evidence aggregation. MAPD achieves the highest success rates across all seven benchmarks for both Qwen3-1.7B and Qwen3-4B. Relative improvement over SDAR is much larger on multi-hop QA (7.9% for 1.7B) than on single-hop QA (2.3%), indicating that the multi-agent design particularly aids complex reasoning. The performance gains are consistent across model scales, with average success rates reaching 39.4% on the 1.7B model and 44.4% on the 4B model.

Removing the structured protocol causes performance to drop below baselines that do not use a proprietary model, showing that raw natural-language trajectories actively harm the student. Adding the multi-agent system without the structured protocol does not improve results and even slightly worsens them, while the protocol alone with a single teacher recovers some gains but still trails the strongest baseline. Only the full MAPD framework combining the structured protocol and multi-agent system achieves the best success rates, demonstrating that both components are essential for effective cross-model distillation. Distilling from raw proprietary model trajectories, with or without a multi-agent system, yields lower average success rates than the GRPO+OPSD baseline that uses no proprietary model at all. Replacing raw text with the structured protocol under a single teacher provides a substantial gain, but this single-agent protocol still underperforms the SDAR baseline. The full MAPD configuration, which pairs the structured protocol with the multi-agent system, outperforms all ablations and baselines on both student model scales.

All three proprietary teachers yield similar average success rates within a 2-point range, confirming that the structured protocol effectively decouples reasoning from teacher-specific style. The OPSD loss weight λ=0.05 gives the best overall performance, while λ=0.10 triggers a trade-off where single-hop accuracy slightly improves but multi-hop performance degrades sharply, indicating a shift toward shallow retrieval over multi-step reasoning. Across Claude, GPT, and Gemini teachers, average success rates vary by less than 2 points (37.9–39.4% on Qwen3-1.7B, 44.0–44.6% on Qwen3-4B), demonstrating teacher-agnostic transfer. Increasing λ from 0.05 to 0.10 raises single-hop scores (e.g., TriviaQA from 62.5 to 63.7 on 4B) but causes multi-hop metrics to drop (2Wiki from 45.7 to 38.4), revealing a shortcut toward retrieval without reasoning.

The evaluation uses Qwen3-1.7B and Qwen3-4B student models on multi-hop question answering benchmarks, comparing MAPD against several baselines. MAPD achieves the best results, especially on complex multi-hop tasks, highlighting the value of multi-agent decomposition and evidence aggregation. Ablation shows that both the structured protocol and multi-agent system are critical, as raw trajectories harm performance and partial configurations underperform. Teacher-agnostic transfer and a moderate OPSD loss weight further confirm the framework's robustness and its ability to balance retrieval with multi-step reasoning.