One-sentence Summary

Researchers from Stanford University, USC, Carnegie Mellon University, and Allen Institute for AI propose REL-A.I., an interaction-centered evaluation framework that measures how much humans depend on LLM outputs rather than relying on simple calibration, finding that contextual features such as perceived competence and knowledge domain significantly shape reliance behavior.

Key Contributions

• This paper introduces REL-A.I., an interaction-centered evaluation framework that measures human reliance on language-model-generated responses rather than only assessing calibration between model probabilities and verbalized confidence.

• Using REL-A.I., the work shows that human reliance varies with interactional context, including prior interactions, model greetings that convey warmth or competence, and the domain of the question, so the same expressions of uncertainty can produce different human decisions.

• Human-participant results show higher reliance for calculation questions by 10% and for models perceived as more competent by 30%, indicating that calibration and language quality alone are insufficient for evaluating human-LLM interaction risks.

Introduction

As large language models are increasingly used in multi-turn, domain-spanning human-AI interactions, a major safety concern is whether users over-rely on model outputs. Prior work has largely focused on verbal calibration, checking whether a model’s stated confidence matches the actual correctness of its answer. The authors argue that this is not enough: a calibrated response can still encourage risky behavior if interaction cues such as greetings, previous confidence levels, or subject matter change how humans interpret the output. To address this gap, they introduce REL-A.I., an evaluation framework that measures in-situ human reliance on LLM-generated responses using epistemic markers, showing that the same expression of uncertainty can lead to different human decisions depending on the surrounding interaction context.

Method

The authors design REL-A.I. as an interaction-centric evaluation framework for measuring human reliance on language model generations. Rather than evaluating a model’s accuracy directly, the method observes the behavioral decision of whether a human chooses to accept an AI agent’s output as useful. The framework combines three modules: a self-incentivized task, epistemic markers extracted from real language model behavior, and meta-level perception questions. Together, these modules create a controlled yet interactive setting where contextual differences can be linked to observable reliance behavior.

At the center of the pipeline is a game-like trivia task. Each participant is presented with a question and the beginning of an AI agent’s response. The response starts with an expression of certainty, uncertainty, or moderate confidence, but the full answer is withheld. The participant then makes a binary choice: rely on the agent or indicate that they will look up the answer themselves. Because the complete answer is not shown, reliance is not driven by visible answer content. Instead, the participant’s behavior is shaped by the epistemic cue, their own knowledge, and the incentive structure of the task.

The task uses explicit point-based incentives to reduce careless participation. A positive score is achieved only when the participant correctly relies on a correct system output. If the participant relies on a wrong system output, they lose points. If they choose not to rely and instead indicate that they will look it up, they receive no penalty and no reward. Formally, let $R_i$Ri​ denote whether the participant relies on the agent in trial $i$i, and let $C_i$Ci​ denote whether the agent’s eventual answer is correct. The trial score can be written as

$$S_i = \begin{cases} 1, & R_i = 1 \text{ and } C_i = 1, \\ -1, & R_i = 1 \text{ and } C_i = 0, \\ 0, & R_i = 0. \end{cases}$$Si​=⎩⎨⎧​1,−1,0,​Ri​=1 and Ci​=1,Ri​=1 and Ci​=0,Ri​=0.​

This scoring rule encourages participants to evaluate whether the agent’s displayed confidence is worth trusting. Reliance rate is then computed for each epistemic expression or experimental condition. If $\mathcal{I}_m$Im​ denotes all trials in which expression category $m$m is shown, the reliance rate can be expressed as

$$\operatorname{Reliance}(m) = \frac{1}{|\mathcal{I}_m|} \sum_{i \in \mathcal{I}_m} R_{i,m},$$Reliance(m)=∣Im​∣1​i∈Im​∑​Ri,m​,

where $R_{i,m}$Ri,m​ equals one when the participant chooses to rely on the agent and zero otherwise.

The second module injects epistemic markers into the agent’s initial response. These markers are not synthetic templates alone; they are based on expressions frequently produced by publicly deployed language models when expressing confidence or uncertainty. During the task, every agent response begins with one such expression, making the model’s stated confidence the main observable cue. The expressions are grouped into three categories: weakeners, strengtheners, and weakened strengtheners. Weakeners communicate uncertainty, such as suggesting that an answer might be true. Strengtheners communicate strong certainty. Weakened strengtheners communicate moderate confidence, such as belief or likelihood rather than absolute certainty.

The framework also includes a repeated-interaction design. The same participant engages with multiple agents or systems within a single session. Each agent may appear under a slightly different interactional context. This design allows separate mental models to form for different agents, while still keeping participant-level behavior comparable. By observing the same person’s reliance decisions across different contexts, the method reduces cross-subject variability and provides a more robust estimate of how contextual features influence reliance.

Finally, after task performance is recorded, the pipeline collects meta-level perception data. Participants rate their perception of the agent along dimensions such as warmth, competence, and humanlikeness using a five-point Likert scale. They also answer a simple yes-or-no question about whether they would be willing to work with the agent again. This component does not replace the behavioral measurement, but it complements it by connecting reliance behavior to broader perceptions of the agent.

Experiment

The study evaluates human reliance on LLM epistemic markers using an in-situ gamified task that presents users with multiple systems and varying interaction contexts, including presentation cues, prior confidence patterns, and subject domain. The experiments find that simple anthropomorphic cues such as greetings increase perceived warmth, competence, and humanlikeness and correlate with higher reliance, with perceived competence as the strongest associated attribute. Reliance also depends on contextual calibration, as the same moderate uncertainty expression is relied on less after a consistently confident model than a less confident one, and reliance is higher in mathematically oriented domains than in non-computational domains. Overall, human reliance is shaped by interactional context rather than epistemic markers alone, suggesting that evaluation and calibration should account for system presentation, user history, and task domain.

The experimental setups compare human reliance across three variables: presentation, prior interaction confidence, and subject domain. Presentation conditions compare a control agent with a greeting agent, while confidence conditions contrast a highly confident model with an unconfident model. The domain conditions cover professional accounting, psychology, and law, with comparable use of strengtheners, weakened strengtheners, and weakeners across those domains. The greeting condition changes presentation while preserving the same interaction mix as the control condition. The confidence manipulation shifts from strengtheners without weakeners to weakeners without strengtheners. The domain manipulation compares accounting, psychology, and law under similar interaction conditions. System and template presentation order is randomized for participants.

Adding greetings to otherwise identical agent responses generally increased perceived warmth, and most greetings also raised perceived competence, humanlikeness, and user reliance. Some greetings produced larger effects than others, while one greeting showed a negligible and nonsignificant change in reliance and another reduced perceived humanlikeness. The patterns suggest that simple conversational cues can shape user trust and reliance, with perceived competence appearing especially connected to reliance changes. Greetings usually increase perceived warmth and often increase perceived competence and reliance. Effects vary by greeting wording, with some greetings producing stronger increases and others showing weaker or mixed outcomes. Perceived competence, warmth, and humanlikeness are linked to greater reliance, while a greeting can also lower perceived humanlikeness.

Weakened strengtheners, such as moderately confident expressions, are relied on less when they come from an agent that previously expressed high confidence than when the same expressions come from an agent that used lower confidence statements. The reduction is reported as statistically significant and appears for several moderate expressions, suggesting that an agent's prior verbalized confidence shapes later reliance decisions. In the compared response types, high-confidence strengtheners receive strong reliance in the confident agent, while low-confidence weakeners receive very low reliance in the lower-confidence agent. Moderate weakened strengtheners are relied on more when paired with an unconfident agent than a confident agent. The difference holds for phrases such as I'm pretty sure, I think, and It's likely. Low-confidence weakeners show very low reliance in the unconfident agent setting.

Human reliance on model answers is strongest in computationally oriented domains, especially mathematics and abstract algebra, and weaker in philosophy, world religion, and law. Stronger confidence expressions tend to elicit higher reliance than weakened confidence expressions, while weakener expressions show the lowest reliance. These trends suggest that domain perception can influence how much people trust model outputs. Mathematics and abstract algebra receive higher reliance than philosophy, religion, and law. Significant increases appear for computational domains, especially among weakened confidence expressions. Reliance decreases as expressed confidence becomes weaker. The results align with the idea that people may overgeneralize machine strength to computation heavy tasks.

The experiments evaluated how human reliance changes when an agent’s presentation style, prior expressed confidence, and response domain are varied. Results show that simple conversational cues such as greetings can increase perceived warmth, competence, humanlikeness, and reliance, though the effect depends on wording. Reliance on moderate confidence statements also depends on prior interaction context, with users trusting them more after lower confidence than after higher confidence. Finally, reliance is highest for computationally oriented domains and decreases with weaker expressed confidence, indicating that perceived machine competence and confidence signals jointly shape trust.