One-sentence Summary

Researchers from University of Washington, UC Berkeley, HKUST, and CMU propose cooperative and competitive multi-LLM collaboration approaches in which LLMs probe each other to identify knowledge gaps and abstain from low-confidence answers, achieving up to $19.3\%$19.3% improvement in abstain accuracy over the strongest baseline across four QA tasks.

Key Contributions

• A taxonomy of 11 abstention baselines is constructed across calibration, training, prompting, and self-consistency methods, repurposing them as AbstainQA mechanisms for identifying LLM knowledge gaps and preventing incorrect answers. This evaluation examines the limits of existing uncertainty and self-reflection approaches.

• Two collaboration-based abstention methods, COOPERATE and COMPETE, are proposed, in which one LLM probes another for missing or uncertain knowledge cooperatively or competitively. These methods promote mechanistic reflection through multi-LLM interaction and target robust abstention in direct QA, retrieval-augmented QA, and multi-hop reasoning.

• Experiments with three LLMs on four QA tasks demonstrate that COOPERATE and COMPETE improve abstain accuracy by up to 19.3% over the strongest baseline, advancing state-of-the-art AbstainQA performance. Further analysis shows the methods can pinpoint failure cases in retrieval-augmented QA and knowledge gaps in multi-hop reasoning.

Introduction

Large language models encode substantial real-world knowledge, but gaps in training data or unreliable internal knowledge can lead to confident hallucinations and biased answers. In question answering, it is therefore critical to detect when a model should abstain rather than generate an incorrect response. Existing abstention methods, such as calibration, training, prompting, and self-consistency approaches, often require held-out data or rely on a single model’s ability to evaluate its own outputs, which can limit generalization and stability. The authors contribute a taxonomy of 11 existing abstention baselines and propose COOPERATE and COMPETE, two multi-LLM collaboration methods that use cooperative or competitive reflection to detect knowledge gaps more robustly.

Method

The authors formulate LLM abstention as a reliability estimation problem over a generated answer. Given a question $\pmb{q}$q and the model answer $\pmb{a}=\mathrm{LLM}(\pmb{q})$a=LLM(q), the goal is to decide whether to accept $\pmb{a}$a or reject it and abstain. A heldout set $(\pmb{q},\bar{\pmb{a}})\in\mathcal{H}$(q,aˉ)∈H is used to tune thresholds or classification parameters so that the abstention policy minimizes erroneous accept and reject decisions.

One class of methods relies on calibration. In token probability based abstention, the LLM produces a confidence score $p(\pmb{a})$p(a) from token probabilities. The threshold $p^*$p∗ is selected on the heldout set by minimizing abstention errors:

At inference time, the model abstains whenever $p(\pmb{a})<p^*$p(a)<p∗. Temperature scaling improves this calibration process by reshaping the answer distribution with a scalar temperature $\tau$τ and optimizing $\tau$τ on the heldout set using cross entropy. After obtaining a calibrated temperature $\tau^*$τ∗, a probability threshold is applied to the temperature scaled output. The authors also consider verbalized confidence, where the LLM first generates an answer and is then prompted to report the probability that the answer is correct. This verbalized score is thresholded in the same way.

Training based abstention methods learn a decision boundary from model states or generated answers. For hidden layer probing, the authors extract the question encoding $\mathbf{e}_{\pmb{q}}=\mathrm{LLM}\text{-}\mathrm{hidden}(\pmb{q})$eq​=LLM-hidden(q) and train a linear probe or feed forward module to predict accept or reject:

The probe is optimized with cross entropy on the heldout set, and the LLM abstains when the predicted label indicates rejection. An external verifier variant trains a separate encoder model to judge the truthfulness of the generated answer from the pair $(\pmb{q},\mathrm{LLM}(\pmb{q}))$(q,LLM(q)), often using the [CLS] representation for classification. Instruction tuning based abstention instead constructs supervised examples from the heldout set. If the generated answer is correct, the pair $(\pmb{q},\pmb{a})$(q,a) becomes a tuning instance. If it is incorrect, the target is replaced with an explicit abstain message. At inference time, generating the abstain message serves as the abstention signal.

Prompting based methods elicit self evaluation without changing model weights. The LLM can be asked to judge its own answer as true or false, to indicate whether additional information is needed, or to first generate an answer freely and then check whether that answer matches a multiple choice option. In these variants, the model is instructed to abstain when the answer is judged false, more information is required, or no option matches the freely generated answer. Consistency based methods add a further signal by checking whether the model repeatedly produces the same answer. For multiple reasoning paths $\{(\pmb{a}_i,\pmb{p}_i)\}_{i=1}^k${(ai​,pi​)}i=1k​, the authors define a plurality index:

This measures how often the most frequent answer appears among the $k$k generated paths. A threshold $\tau^*$τ∗ is tuned on the heldout set, and the model abstains when $\mathrm{plu}(\mathrm{LLM},\pmb{q},k)<\tau^*\cdot k$plu(LLM,q,k)<τ∗⋅k.

To reduce dependence on unstable self evaluation and limited heldout data, the authors further propose a multi LLM collaboration framework. Instead of asking a single LLM to evaluate its own output, multiple LLMs interact to produce external evidence for accept or reject decisions. The framework combines two complementary mechanisms, COOPERATE and COMPETE.

The proposed collaboration pipeline is illustrated below.

In the COOPERATE mode, a question $\pmb{q}$q is first answered by a primary LLM, yielding $\pmb{a}=\mathrm{LLM}(\pmb{q})$a=LLM(q). Several expert LLMs then provide natural language feedback $\{f_1,\ldots,f_k\}${f1​,…,fk​} about the proposed answer. These feedback signals are summarized by a judge LLM, which produces the final abstention decision:

The feedback can be generated in two modes. In the self mode, the same LLM is prompted to specialize in different domains by first generating a domain specific knowledge passage about $\pmb{q}$q, then acting as a reviewer to assess the proposed answer. In the others mode, separate LLMs independently generate feedback $\pmb{f}_i=\mathrm{LLM}_i(\pmb{q},\pmb{a})$fi​=LLMi​(q,a). This design leverages differences in model knowledge coverage so that gaps in one model’s internal knowledge can be identified by another model’s feedback.

In the COMPETE mode, abstention is driven by answer stability under conflicting information. Starting from the original answer $\pmb{a}=\mathrm{LLM}(\pmb{q})$a=LLM(q), other LLMs generate alternative answers $\{\pmb{a}'_1,\ldots,\pmb{a}'_k\}${a1′​,…,ak′​} along with supporting knowledge passages. The original LLM is then prompted again with each conflicting knowledge passage to produce a revised answer:

If $\tilde{\pmb{a}}_i$a~i​ remains equal to the original answer, the model demonstrates consistency. If it changes, the model has been influenced by the conflicting input, which signals lower confidence. The process is repeated across multiple alternative answers, and the LLM abstains when the answer changes in a majority of trials.

Experiment

The experiments evaluate LLM abstention across three models and four knowledge-intensive tasks using measures of answer reliability, coverage, abstention quality, and calibration. The results show that collaboration-based abstention methods generally outperform existing baselines and remain robust across diverse domains, reasoning contexts, and model strengths. Additional analyses show that these methods handle absolute abstention scenarios, selective retrieval, multihop reasoning localization, and calibrated abstention scores, while COMPETE emphasizes conservative reliability and COOPERATE offers a more balanced coverage.

Collaboration based abstain strategies, COOPERATE and COMPETE, show the strongest overall reliable accuracy across the four tasks and three language models. Calibration, training, and consistency methods generally outperform simple prompting, while calibration and stronger base models are associated with better abstention quality. COOPERATE and COMPETE give the best reliable accuracy in most evaluated settings over prior baselines. COOPERATE benefits from stronger base models, suggesting its multi step feedback process needs higher capability LLMs. COMPETE strongly favors reliability by abstaining except in highly confident cases, leading to high reliable accuracy. Instruction tuning and self consistency threshold are strong baselines but depend on held out data for tuning. Abstention quality tends to track base model utility, with stronger LLMs achieving better average abstain accuracy.

The results compare abstain strategies on MMLU with Mistral-7B under no retrieval and retrieval settings. Retrieval notably weakens ASK CALIBRATE, while collaboration-based abstain behavior appears more stable when retrieved context is prepended. COMPETE shows strong answer quality without retrieval, supporting its use in selective retrieval decision making. ASK CALIBRATE loses much of its abstain performance when retrieved context is added. COOP-SELF changes only modestly with retrieval, suggesting robustness to prepended context. COMPETE has the strongest answer F1 among the no retrieval variants shown.

With Mistral-7B, abstention calibration error changes substantially across tasks and methods, showing that calibration depends on the approach and dataset. ASK CALIBRATE is comparatively strong on MMLU and Hellaswag, while COOP-SELF and VERIFIER provide the best calibration on their respective strongest tasks. ASK CALIBRATE has the lowest abstain calibration error on MMLU and Hellaswag. COOP-SELF is strongest on K-Cross, and VERIFIER is strongest on Propaganda, indicating dataset-specific calibration tradeoffs.

The experiments evaluate abstention strategies across multiple tasks and language models, comparing collaboration-based, calibration, training, consistency, and prompting baselines. Collaboration-based methods, especially COOPERATE and COMPETE, generally provide the strongest reliable accuracy, with COMPETE being highly conservative and COOPERATE benefiting from stronger base models. Retrieval-focused MMLU results show that retrieved context can weaken some calibrated methods while collaboration-based strategies remain more stable, and calibration quality varies by method and dataset. Overall, reliable abstention depends on both strategy design and base model capability, with collaboration-based approaches offering the most robust performance.