One-sentence Summary

Researchers from HKUST, Tencent, and HKUST(GZ) propose DeepSearch-Evolve, a self-distillation framework that iteratively generates, filters, mixes, and fine-tunes trajectories in the verifiable DeepSearch-World environment without relying on stronger teacher models, enabling the DeepSearch-World-9B agent to achieve competitive results on BrowseComp (31.2%), GAIA (61.5%), and HotpotQA (93.4%), thereby demonstrating scalable self-evolution for long-horizon web agents.

Key Contributions

• DeepSearch-World is introduced as a deterministic, verifiable offline environment for deep search agents, providing reproducible observations, entity-level progress verification, and 420K multi-hop QA tasks over Wikipedia.
• Scaffold process supervision is proposed, where a teacher agent explicitly tracks progress, evidence, failed attempts, and recovery, and these scaffold trajectories are converted into ReAct-format training data to inject planning, memory tracking, grounded reflection, and failure recovery into the student.
• DeepSearch-Evolve, an iterative self-distillation framework, generates and filters trajectories using verified process signals, then fine-tunes the agent, enabling improvement without distillation from stronger models. The resulting DeepSearch-World-9B achieves 31.2% on BrowseComp, 61.5% on GAIA, and 93.4% on HotpotQA.

Introduction

The authors address the challenge of enabling LLM-based tool-use agents to autonomously improve from their own interactions, a key requirement for scalable self-evolving agents that plan, search, browse, and reason with tools. Prior self-improvement pipelines suffer from saturation when fine-tuning on static positive trajectories, from sparse trajectory-level rewards in RL that offer no fine-grained guidance, or from unreliable dense supervision in on-policy self-distillation, because the teacher’s step-level distribution is noisy in tool-use settings. To overcome these, the authors introduce DeepSearch-World, a deterministic and verifiable offline environment built on Wikipedia with entity-level progress verification, and DeepSearch-Evolve, a framework that captures scaffold process supervision (planning, memory, failure recovery) and converts it into ReAct-format data for iterative student training. This allows open-source agents to improve from verified tool-use experience without relying on synthetic trajectories from stronger proprietary models.

Dataset

The authors build a large-scale multi-hop QA dataset and a corresponding offline Wikipedia environment to support scalable self-distillation.

• Dataset composition and sources

  • A QA pool of 420K instances sampled from entity-level random walks on the Wikipedia hyperlink graph. Each instance obfuscates explicit entity mentions, requiring the agent to recover the target entities through reasoning, search-query revision, and evidence discovery.
  • An offline corpus C of roughly 10 million Wikipedia entries (title, caption, and full article text) crawled from the union of all target entities appearing in the QA pool. This ensures all required evidence is self-contained and retrievable.
  • A held-out validation split DeepSearch-Val with 377 high-quality instances, verified by a five-expert pool (at least three experts per instance). These instances are excluded from trajectory generation and model training.

• Key details per subset

  • QA pool (420K instances): Built from H-hop random walks on the Wikipedia graph; questions are automatically generated by masking entity mentions. No explicit filtering is applied beyond the random walk sampling, yielding a broad distribution of reasoning paths, failure modes, and recovery patterns.
  • Offline corpus (~10M entries): Crawled Wikipedia pages for the target entities of the QA pool. Each entry is a structured record with title, caption, and article text. The corpus is deterministic and static, serving as the environment’s knowledge base.
  • DeepSearch-Val (377 instances): Filtered from the QA pool to include only instances whose evidence is fully covered by the offline corpus and retrievable through the provided search and visit tools. Answers are independently validated by multiple experts, making it suitable for reliable behavioral analysis and validation.

• How the data is used

  • The full QA pool (minus DeepSearch-Val) is used to instantiate the DeepSearch-World environment. The agent generates trajectories within this deterministic offline setting, and those trajectories are used for self-distillation training.
  • No explicit training/validation split ratios beyond the 377 held-out instances are mentioned; the remaining ~420K instances constitute the training pool for trajectory generation.
  • The offline corpus is the static knowledge base; the agent interacts with it via search and visit tools, but the corpus itself is not split.

• Processing and metadata construction

  • Random walks of length H are sampled from the Wikipedia hyperlink graph. Each walk defines a chain of target entities; the question is formed by obfuscating the explicit entity mentions, turning the walk into a multi-hop reasoning task.
  • For each QA instance, the target entity set is identified, and the corresponding Wikipedia pages are crawled to build the offline corpus. The corpus entries are structured with title, caption, and full article text.
  • The validation split is curated by selecting instances where the required evidence is covered by the corpus and reachable via the tools, followed by expert answer verification (at least three experts per instance). No further cropping or data augmentation is mentioned.

Method

The authors present DeepSearch-Evolve, a self-distillation framework for search agents designed to improve through iterative self-evolution. The system comprises a verifiable tool environment, a scaffold teacher agent for high-quality trajectory generation, and an iterative training loop.

As shown in the framework diagram:

Verifiable Tool Environment and Data Construction To support scalable self-distillation, the authors construct DeepSearch-World, a deterministic offline Wikipedia environment. They sample entity-level random walks $\tau = (v_1, \dots, v_H)$τ=(v1​,…,vH​) from the Wikipedia hyperlink graph $\mathcal{G} = (\mathcal{V}, \mathcal{E})$G=(V,E) to define H-hop reasoning chains. This process yields 420,000 multi-hop QA instances where explicit entity mentions are obfuscated, forcing the agent to recover targets through reasoning and search. The environment is grounded in a local corpus of approximately 10 million Wikipedia entries, ensuring all required evidence is searchable and verifiable.

The environment exposes two offline tools aligned with standard web agent workflows. The search tool, web_search_wiki, functions as a deterministic retrieval function using a BM25 index to return page titles, captions, and URLs. The visit tool, visit_wiki, maps URLs to full article text via a SQLite index.

To facilitate learning from failures, the environment implements environment-grounded reflection. For each question, a ground-truth entity set is stored. During rollout, if a tool response matches an unresolved entity, the progress is verified. Failed calls trigger staged rule-based reflection, providing generic revision signals initially and stronger guidance, such as canonical names, upon repeated failures. This creates grounded search, fail, reflect, and retry trajectories.

Scaffold Teacher Agent To generate high-quality supervision data, the authors employ a scaffold teacher agent that operates in three phases: Plan, Act, and End. This scaffold explicitly supervises planning, memory, and error recovery during trajectory generation.

As shown in the figure below:

In the Plan phase, given a question $q$q, the teacher initializes a structured progress state $s_0$s0​ containing fields for completed subgoals, pending actions, lessons from failures, and extracted evidence. In the Act phase, the teacher performs interaction steps up to a maximum budget. At step $t$t, it selects a tool call $a_t$at​, receives an observation $o_t$ot​, and updates the state via $s_{t+1} = \mathcal{U}(s_t, a_t, o_t, r_t)$st+1​=U(st​,at​,ot​,rt​), where $r_t$rt​ is the environment-grounded reflection. In the End phase, the teacher generates a concise answer grounded in the verified working memory.

An example of this process illustrates how the teacher decomposes a fuzzy multi-hop question, alternates between tools, updates working memory with verified evidence, and uses grounded reflection to recover from failed searches.

As shown in the figure below:

Self-Evolving Training Loop DeepSearch-Evolve utilizes an iterative loop where the current model $\pi_{\theta_R}$πθR​​ acts as the teacher to generate trajectories, which are then used to train the student $\pi_{\theta_{R+1}}$πθR+1​​.

First, trajectories are generated and verified. The authors retain trajectories with correct answers and apply trajectory-level filtering to remove redundant evidence and inconsistent reasoning.

Next, a scaffold-to-ReAct conversion occurs. Since the goal is to train a deployable ReAct agent independent of external planning prompts, the scaffold structure is removed. The progress states and reflections are rewritten into the assistant reasoning trace. For each tool-use step $t$t, the target thinking block is constructed as:

Here, $P_t$Pt​ summarizes the progress state, including completed subgoals, pending targets, failed-search experience, and verified evidence. $R_t$Rt​ rewrites the environment reflection as self-correction, and $A_t$At​ preserves the rationale for the action $a_t$at​.

Finally, the model undergoes a supervised self-distillation update. The authors adopt evolving SFT for stability in long-horizon tool use. Given a verified trajectory $\tilde{\tau} = (x, y_{1:T})$τ~=(x,y1:T​), the SFT loss minimizes the negative log-likelihood:

This approach samples scaffold rollouts from the current agent, filters them, and converts them into ReAct traces to supervise planning and tool invocation without relying on per-token guidance over unverified states.

Experiment

The evaluation covers seven deep search and reasoning benchmarks, comparing the self-evolving DeepSearch-World-9B against proprietary and open-source agents under a training setup that iteratively generates, filters, and distills environment-verified trajectories. The model, optimized solely from its own verified rollouts, achieves competitive performance and consistently improves over its base backbone, exhibiting extended multi-step interaction, stronger evidence grounding, and more effective tool-use strategies. Analyses reveal that a large and diverse data pool, coupled with rejection sampling and quality filtering, is critical for stable self-improvement, while reflection rewriting and state internalization are key to converting scaffolded trajectories into high-quality training data. These results suggest that deterministic environment verification can replace external teacher supervision, enabling robust agentic skill transfer through iterative self-distillation.

DeepSearch-World-9B, optimized entirely from its own environment-verified rollouts, substantially outperforms its base model and reaches competitive open-source performance on deep search benchmarks. Gains of 23.8 to 48.1 points across six benchmarks show that self-distillation from verified trajectories can learn transferable tool-use behaviors, though English-only training limits cross-lingual transfer. The model also engages in much longer, more grounded tool-use loops, averaging 18.0 rounds versus 4.7 for the base model. DeepSearch-World-9B improves over Qwen3.5-9B-Instruct by 23.8 points on BrowseComp and 37.6 on GAIA, without requiring stronger teacher models or synthetic pipelines. It sustains 18.0 tool-use rounds on average, whereas the base model terminates after only 4.7 rounds, and makes 5.4 visit calls versus 0.9, indicating stronger evidence grounding. The advanced capability score, covering planning, memory, reasoning, self-correction, and evidence gathering, rises from 19% for the base model to 70% for DeepSearch-World-9B.

Removing both rejection sampling and quality filtering yields the lowest SearchQA score. Rejection sampling that selects only answer-correct trajectories provides the dominant gain, while trajectory quality filtering adds a smaller improvement. Combining both techniques achieves the highest accuracy, outperforming alternative methods like SDAR and Skill-SD, demonstrating that verified trajectory filtering is a more effective self-distillation strategy. Rejection sampling alone raises SearchQA substantially over the unfiltered baseline, confirming that answer-correctness verification is the key safeguard for self-distillation. Adding trajectory quality filtering on top of rejection sampling further improves performance, surpassing SDAR and Skill-SD by filtering redundant, weakly aligned, or inconsistent correct-answer traces.

Reflection rewriting is the dominant factor in converting trajectories into training data: removing it drops performance below vanilla SFT, while state internalization adds a smaller but consistent improvement. The full pipeline combining both delivers a clear gain over standard SFT, demonstrating the value of carefully processed self-distillation traces. Reflection rewriting is critical: without it, performance falls to 16.7%, even lower than vanilla SFT, because raw reflections contain artifacts like [REFLECTION] tokens that distort the reasoning distribution. State internalization provides a consistent but modest boost, adding planning, memory, and progress tracking information; removing it reduces the full pipeline's performance from 31.9% to 23.5%. The full conversion pipeline substantially outperforms vanilla SFT, improving DeepSearch-Val from 25.0% to 31.9% and showing the benefit of evolving self-distillation with refined trajectory processing.

The study evaluates DeepSearch-World-9B, a model improved via self-distillation on environment-verified trajectories, against its base model on deep search benchmarks, showing gains of 23.8 to 48.1 points and sustained tool-use loops averaging 18.0 rounds versus 4.7. Ablation experiments reveal that rejection sampling of answer-correct trajectories is the dominant improvement factor, while trajectory quality filtering adds a secondary boost, and that reflection rewriting is critical to avoid degrading performance, with state internalization providing a modest further gain. Overall, the full pipeline combining careful filtering and trajectory processing substantially outperforms vanilla SFT, demonstrating that refined self-distillation traces can teach transferable evidence-gathering and reasoning behaviors.