One-sentence Summary

Researchers from the University of Leeds, Southwest Jiaotong University, University of Washington, and Allen Institute for AI introduce Agentic Abstention, a sequential decision problem of when to stop acting under uncertainty, evaluate 13 LLM-as-agent systems and 2 agent scaffolds on over 28,000 tasks, and propose CONVOLVE, a context engineering method that distills interaction trajectories into stopping rules, raising Llama-3.3-70B's timely recall rate from $26.7$26.7 to $57.4$57.4 on WebShop.

Key Contributions

• Agentic Abstention is introduced as a sequential decision problem in which a tool-using agent must choose at each turn to answer, abstain, or gather more information when facing uncertainty.
• A benchmark of over 28,000 tasks spanning web shopping, terminal environments, and question answering evaluates 13 LLM-as-agent systems and 2 agent scaffolds, showing that timely abstention is a persistent challenge: all tested models average below 40% timely recall.
• The context-engineering method CONVOLVE distills full interaction trajectories into a reusable playbook of stopping rules, improving timely abstention without parameter updates, e.g., raising Llama-3.3-70B’s timely recall from 26.7% to 57.4% on WebShop.

Introduction

The authors examine a fundamental weakness in tool-using agents: the failure to recognize when a task is infeasible or ambiguous and to stop acting rather than engaging in wasteful exploration. Prior work has largely concentrated on improving task completion via better planning, memory, and agent scaffolds, with recent efforts touching on uncertainty and clarification but never studying abstention as a sequential, environment-driven decision. Agents often persist in interaction even when success is impossible, especially when a task appears solvable initially and only reveals its infeasibility after multiple actions. To fill this gap, the authors formalize Agentic Abstention, where an agent must decide among acting, answering, or abstaining as context evolves. They construct a benchmark of over 28,000 instructions across web shopping, terminal use, and QA, evaluating 13 LLM-as-agent systems plus two scaffolds. The results show that timely abstention is a severe challenge, with the best configurations achieving less than 40% timely recall on average. They then introduce CONVOLVE, a context-engineering method that learns reusable stopping rules from full interaction trajectories and provides them as in-context guidance. On web shopping, CONVOLVE lifts timely recall from 26.7% to 57.4% and overall recall from 83.2% to 100.0% without any parameter updates.

Dataset

The authors build evaluation datasets for agentic abstention across three representative scenarios. All sets are used solely for assessing the model’s ability to recognize when it should not attempt a task, and none involve training splits.

• Web-based decision‑making (WebShop adaptation)

• Source: first 500 instructions from the WebShop test set.

• Solvable subset: the original 500 solvable tasks kept unchanged.

• Unverifiable subset: 500 newly constructed abstention‑warranted tasks, split into two types:

• Request‑based abstention (249 tasks): the original instruction is rewritten by an LLM into three forms – Subjective Preference, Underspecified Intent, or False Premise/Contradiction. All rewrites are manually reviewed, and unnatural or ambiguous cases are filtered.

• Environment‑based abstention (251 tasks): the original instruction is left intact, but the target items are removed from the product catalog and the Lucene search index is rebuilt so that no matching product exists. These are called Missing Target tasks.

• Final size: 1,000 instances with a balanced 1:1 solvable/unsolvable split.

• Terminal‑based task execution (Terminal‑Bench 2.0 adaptation)

• Source: all 89 tasks from Terminal‑Bench 2.0, each with a natural‑language instruction, Dockerized environment, and verification tests.

• Solvable subset: the original 89 tasks kept unchanged.

• Unverifiable subset: 188 newly constructed abstention‑warranted tasks:

• Request‑based abstention (167 tasks): instructions are rewritten into False Premise/Contradiction (87 tasks) or Underspecified Intent (80 tasks) using a dual‑agent correction pipeline. A rewriting agent proposes an adapted instruction, a validation agent checks it, and invalid rewrites are revised for up to three rounds. All final instructions are manually reviewed.

• Environment‑based abstention (21 tasks): environments are manually modified to remove a prerequisite (e.g., a file, dependency, permission, or service) needed for completion; these are Missing Prerequisite tasks.

• Final size: 277 instances (89 solvable, 188 unsolvable).

• Interactive QA (AbstentionBench adaptation)

• Source: AbstentionBench, a multi‑dataset benchmark for answer‑or‑abstain decisions. The authors retain 16 of its datasets, excluding those where questions could often be resolved by search.

• Composition: 27,073 samples that include both answerable questions and abstention‑warranted questions spanning five categories: Answer Unknown, False Premise, Subjective, Underspecified Context, and Underspecified Intent.

• Processing: each QA instance is converted into a sequential decision problem. The agent can answer, abstain, or issue a search query. To ensure reproducibility, retrieval is performed against the enwiki‑20260101 Wikipedia dump, returning the top‑3 documents per call, with a maximum of 10 search calls allowed. This differs from the original single‑turn static‑prompt setting.

The authors use these datasets only for evaluation. The web and terminal sets test the model’s ability to recognize infeasibility from the request alone or after interacting with the environment, while the QA set examines sequential decision‑making with search. No training data or mixture ratios are described because the paper focuses on evaluation.

Method

The authors define Agentic Abstentionas an agent's ability to recognize when a task is infeasible and to abstain rather than answering incorrectly or taking unnecessary actions. This concept is formulated as a partially observable Markov decision process (POMDP), denoted by $\mathcal{M} = (S, \mathcal{A}, \mathcal{O}, T, \Omega, R)$M=(S,A,O,T,Ω,R). Here, $S$S represents the latent task state, including unobservable properties such as whether the task is resolvable given the available context and tools. The action space is defined as $\mathcal{A} = \{\text{ANSWER}, \text{ABSTAIN}, \text{ACT}\}$A={ANSWER,ABSTAIN,ACT}, where ANSWER denotes a terminal task-completion action, ABSTAIN denotes a terminal decision to stop under the current information state, and ACT denotes a non-terminal external action to gather additional information. At each turn $t$t, the agent receives an observation $o_t \in \mathcal{O}$ot​∈O and selects an action based on the interaction history $h_t$ht​ using a history-dependent policy $\pi(a_t \mid h_t)$π(at​∣ht​). If the agent chooses ACT, the interaction continues; if it chooses ANSWER or ABSTAIN, the episode terminates. Requests for user clarification are also treated as instances of ABSTAIN.

To improve agentic abstention, the authors propose CONVOLVE (Context Evolution), a framework that learns from multi-step interaction trajectories. Rather than learning solely from final answers, CONVOLVE analyzes full environment rollouts and distills them into reusable stopping rules that are added to the agent's context for future episodes. Let $c^{(k)}$c(k) denote the evolving context available after the $k$k-th episode. For each training episode, the agent interacts with the environment to produce a trajectory $\tau^{(k)} = (x^{(k)}, o_1^{(k)}, a_1^{(k)}, \dots, o_{T_k}^{(k)}, a_{T_k}^{(k)})$τ(k)=(x(k),o1(k)​,a1(k)​,…,oTk​(k)​,aTk​(k)​). After the episode, a reflection agent analyzes the trajectory to derive episode-level feedback $y^{(k)} = \phi(\tau^{(k)})$y(k)=ϕ(τ(k)), capturing abstention-relevant signals. The context is then updated as $c^{(k+1)} = \mathcal{U}(c^{(k)}, \tau^{(k)}, y^{(k)})$c(k+1)=U(c(k),τ(k),y(k)), where $\mathcal{U}$U is the context update operator.

The authors instantiate CONVOLVE on an abstention-only subset of WebShop, but the framework relies on carefully constructed abstention-warranted tasks across different environments. For terminal-based task execution, each adapted task comprises a containerized environment, an instruction, verification tests, and a reference solution. The original instructions are rewritten to create infeasible requests, such as those containing false premises or underspecified intents.

As shown in the figure below:

This adaptation process yields tasks where abstention becomes the correct action either immediately from the request or after interacting with the environment. During the CONVOLVE training process, environment rollouts are conducted with the learned playbook appended to the system prompt. After each rollout, a reflection model reviews the full trajectory to identify signals, such as whether the agent continued acting after the task became infeasible. A curator model then converts this reflection into concise playbook updates, organized into fixed sections to keep the evolving context structured. To prevent context overflow, the curator input is deterministically truncated, and the playbook budget is capped. The framework is evaluated on a held-out set using the same model used for rollouts, comparing performance against a baseline without playbook injection.

Experiment

The evaluation assesses LLM-based agents in web, terminal, and QA scenarios, measuring their ability to abstain from infeasible tasks with metrics like abstention recall (AbsRec@K) and over-abstention rate. Timely abstention proves difficult across all settings, with most models failing to recognize infeasibility early and performance varying widely by abstention category and agent scaffold. Model scale improves eventual recall but not promptness, while stronger reasoning can mitigate over-abstention without reliably boosting timely recognition. A context-engineering approach, CONVOLVE, significantly enhances abstention using only 20 training trajectories, and lessons from smaller models transfer effectively to larger ones.

CONVOLVE shows strong data efficiency, using only 20 trajectories to improve agentic abstention on WebShop. In-context learning helps larger models but can hurt smaller ones, while lessons learned by a small model transfer effectively to a large model, almost matching self-play gains. A playbook learned by Llama-3.3-70B raises Llama-3.3-8B timely recall from 6.9 to 12.9 while preserving overall recall and SPL. In-context learning for the 8B model lifts timely recall but causes a 10.0-point drop in overall recall, showing a reliability trade-off. Transferring lessons from an 8B model to a 70B model boosts timely recall from 26.7 to 55.3 and overall recall from 83.2 to 99.0, close to the 70B self-learned result of 57.4 / 100.0. All trained variants rely on only 20 interaction trajectories, demonstrating high data efficiency.

CONVOLVE demonstrates strong data efficiency, requiring only 20 trajectories to learn a playbook. In-context learning proves beneficial for larger models but can harm smaller ones, whereas transferring a learned playbook from a small model to a large model nearly matches the large model's own self-play gains. A large model's playbook lifts a small model's timely recall with minimal impact on overall recall, avoiding the reliability trade-off imposed by in-context learning. Overall, this cross-model transfer achieves substantial improvements while preserving data efficiency.