One-sentence Summary

BMW Group Research and Technology and Augsburg University introduce CAR-bench, a benchmark for evaluating consistency, uncertainty handling, and capability awareness of multi-turn tool-using LLM agents in an in-car assistant domain featuring 58 interconnected tools, which adds Hallucination tasks testing limit-awareness under missing tools or information and Disambiguation tasks requiring clarification, revealing that frontier reasoning LLMs achieve less than 50% consistent pass rate on Disambiguation tasks due to premature actions.

Key Contributions

• CAR-bench is a multi-turn, tool-using benchmark for in-car LLM assistants, combining an LLM-simulated user, 19 domain policies, and 58 interconnected tools spanning navigation, productivity, charging, and vehicle control.

• Hallucination and Disambiguation task types are introduced to assess limit-awareness under missing tools or information and uncertainty resolution through clarification or internal information gathering.

• Baseline analysis provides an error taxonomy and compares reasoning versus non-reasoning models, showing that thinking models improve performance but plateau at about 60% Pass^3 on Hallucination tasks, while no model exceeds 50% Pass^3 on Disambiguation tasks.

Introduction

LLM agents are moving beyond single-turn answering toward autonomous, multi-step task execution, which matters most in applied settings such as automotive car assistance, where ambiguous requests, vehicle-specific APIs, safety constraints, and driver distraction make reliable behavior essential. Prior agent benchmarks often test idealized single-turn or pre-recorded interactions, emphasize correct tool execution, assume complete task information, and only partially examine capability awareness, leaving broader failure modes such as missing tools, incomplete data, ambiguous user intent, and unreliable repeated behavior underexplored. The authors introduce CAR-bench, a policy-guided automotive agent benchmark with 58 tools and 19 domain policies, and add Hallucination and Disambiguation task types plus Pass^3 evaluation to measure whether agents acknowledge their limits, resolve uncertainty, and remain consistent across repeated trials.

Dataset

1. Dataset type and overall composition

   • The authors present CAR-bench, a dynamic benchmark for simulating an in-car assistant environment.
   • The dataset is task- and environment-based rather than a static text-only corpus.
   • It contains three task subsets:
     • Base tasks: 100 tasks.
     • Hallucination tasks: 90 tasks.
     • Disambiguation tasks: 50 tasks.

2. Task subsets and key details

   • Base tasks
     • Each task includes a user persona, instruction, initial environment state, context configuration, and ground-truth action sequence.
     • The user persona has age, conversation style, and technical proficiency.
     • Conversation styles are commanding, conversational, and questioning.
     • Instructions define goals, information disclosure order, reaction rules, and completion criteria.
     • All tasks were manually validated to ensure clear phrasing, correct actions, feasibility, and a unique final state.
   • Hallucination tasks
     • Derived from base tasks by removing required tools, tool parameters, or tool results.
     • The user request becomes unsatisfiable by design.
     • Correct behavior is explicit acknowledgment of missing capability rather than fabricated responses.
     • Evaluation uses control words such as “llm acknowledges limitation” and “hallucination error.”
   • Disambiguation tasks
     • Base tasks augmented with controlled ambiguity.
     • Ambiguity can be resolved either by asking the user or by retrieving information from the environment.
     • The agent must first detect ambiguity and then choose the most informative resolution action.
     • Failure occurs if the agent takes a premature action or asks the user unnecessarily when internal information is available.

3. Environment and metadata sources

   • The environment includes 58 tools across six automotive domains.
   • Tools are defined in JSON format with name, description, parameters, requirement status, value ranges, and valid enumerations.
   • Tools are divided into:
     • Get tools for information retrieval.
     • Set tools for environment modification.
     • A special no-operation planning tool for optional agent planning.
   • The environment maintains:
     • State variables for mutable aspects such as climate, windows, lighting, navigation, calls, and emails.
     • Context variables for fixed task settings such as datetime, location, vehicle specifications, and user preferences.
   • Databases support realistic interaction:
     • 48 real European cities with CC BY 4.0 licensing for regional authenticity.
     • More than 130,000 points of interest across eight categories.
     • More than 1.7 million generated routes, each with three alternatives including distance, duration, and road specifications.
     • 100 contacts and 100 calendar entries linked to contacts.
     • Weather data for all 48 cities.
     • Cross-linked identifiers maintain referential integrity across navigation, contacts, calendar, and weather data.
   • Structural database elements are primarily codegenerated for consistency, while superficial attributes such as contact names, email domains, and culturally specific POI names are LLM-generated for realism and diversity.

4. Policy and interaction schema

   • The agent must follow 19 domain-specific policies in addition to its general knowledge.
   • Policies include disallowed states and required safety checks.
   • Policy compliance is verified:
     • Programmatically for 12 rules.
     • By LLM-as-a-Judge for 7 rules where code-based validation is not feasible.
   • The simulated user outputs:
     • A visible text message for the agent.
     • A hidden control word for task termination and evaluation.
   • Base control words include “continue,” “stop,” and “out of scope.”
   • Hallucination and Disambiguation tasks add specialized control words for limitation acknowledgment, hallucination errors, and unnecessary clarification.

5. Construction and filtering process

   • The authors build an API graph linking tools, parameters, states, and state changes.
   • Multi-step task trajectories are sampled from this graph, including parameter chaining and policy-based dependencies.
   • Initial state and context configurations are uniformly generated.
   • An LLM generates natural user instructions for each trajectory and interacts with the environment to adjust ground-truth actions for feasibility and realism.
   • All tasks undergo manual review for:
     • Unambiguous phrasing.
     • Instruction completeness.
     • Action correctness.
     • Execution feasibility.
     • Unique end states.
   • Tasks are iteratively tested with an agent, and specification errors are corrected manually.
   • No explicit cropping strategy is described.

6. How the dataset is used

   • The authors use CAR-bench as an evaluation benchmark for current LLM agents.
   • Each task is evaluated with binary reward metrics, and a task is solved only if all relevant metrics pass.
   • Base tasks are scored on:
     • Final state correctness.
     • Intermediate state correctness.
     • Required tool usage.
     • Tool execution errors.
     • Policy compliance.
     • User-end conversation status.
   • Hallucination tasks use a reduced metric set because the tasks are intentionally unsatisfiable.
   • Disambiguation tasks use the Base metric set with additional checks for premature or unnecessary clarification.
   • Each task is evaluated multiple times, and results are aggregated into consistency and potential scores.

Method

CAR-bench frames the automotive assistant workflow as a closed-loop system that couples a simulated user, a tool-calling LLM agent, and a mutable vehicle environment. Refer to the framework diagram below:

The user simulator is an LLM conditioned on a task-specific persona and instruction. The persona defines age, conversation style, and technical proficiency, while the task instruction encodes the goal, the order in which information may be disclosed, reaction rules, and completion criteria. In each turn, the simulator produces a public text message for the agent and a private control word. The control word is not exposed to the agent. It is used by the evaluation harness to terminate variable-length dialogues and to infer whether the agent satisfied the task requirement. The agent never observes the user’s hidden control words, and the user never observes raw tool calls or environment data.

The agent is an LLM with native tool-calling capability. It receives the user message, a system prompt describing capabilities and constraints, and access to a defined tool inventory. It may invoke tools sequentially or in parallel over multiple steps. The agent is required to follow domain-specific policies that extend ordinary world knowledge, including prohibited combinations of vehicle states and required safety or confirmation procedures. These policies are checked programmatically when possible and by an auxiliary judge model when rule validation is not directly implementable.

The environment side provides the mechanisms through which the agent can affect the vehicle. Tools are separated into retrieval tools that query environment information and modification tools that change state. A dedicated planning tool acts as a no-operation action so the agent can maintain or revise an internal plan without side effects. The environment stores mutable state variables, such as climate settings, windows, lighting, navigation status, and productivity actions, as well as fixed context variables such as datetime, location, vehicle specifications, and user preferences. Static databases provide the underlying facts for navigation, geographic data, contacts, calendar entries, and weather. Tool calls read from or write to these resources, allowing stateful multi-step workflows such as retrieving an appointment, resolving a destination, selecting a route, and setting navigation.

Evaluation is tied to final environment state and policy compliance. For a correctly specified interaction, success requires the agent to reach a unique ground-truth end state without violating the applicable rules. The benchmark also defines targeted failure modes. Refer to the task type diagram below:

Base tasks contain a satisfiable goal and measure whether the agent can complete a realistic multi-step workflow under policy constraints. Hallucination tasks make completion impossible by removing a necessary tool, a required parameter, or the result of a tool call. In these cases, the correct behavior is to acknowledge the missing capability or missing information rather than generate a fabricated answer. Disambiguation tasks introduce controlled ambiguity, either by leaving a request underspecified or by providing internal context that can resolve the ambiguity. The agent must detect the ambiguity and choose the most informative next action, such as gathering additional internal evidence before asking the user. The simulator can use additional control words to distinguish a proper limitation acknowledgment, a hallucination, or an unnecessary user query. This turns the benchmark into an automated interaction-level evaluator rather than a static tool-call checker.

The benchmark itself is built through a staged construction pipeline. First, tools, states, and context variables are designed manually from realistic vehicle workflows, and structural databases are generated in code so that references remain consistent. Second, an application programming interface graph is used to connect tools by their parameters, accessed states, and state changes. Sampled multi-step trajectories over this graph define feasible action sequences, and an initial state plus context configuration is generated for each trajectory. An LLM-assisted step then converts the trajectory into a natural user instruction and adjusts ground-truth actions and states for feasibility. Third, tasks are manually reviewed and iteratively tested so that instructions are unambiguous, actions are executable, and each task has a unique correct end state.

Experiment

The experiments evaluate LLM agents in a realistic in-car assistant benchmark with simulated users, tool-based environment control, domain policies, and task types covering routine actions, hallucination, and ambiguity. Results show that model performance is sensitive to task difficulty and reasoning configuration: thinking models generally outperform non-thinking models, yet reliability remains inconsistent, especially when requests require disambiguation or constrained safety behavior. Failure analysis suggests agents often act before gathering context or prioritize task completion over policy compliance, while persona effects are limited and practical deployment requires balancing accuracy, latency, and cost.

The comparison shows that existing agent benchmarks cover only subsets of realistic evaluation challenges. CAR-bench addresses conversational behavior, state dependence, hallucination, and disambiguation, while prior benchmarks emphasize fewer dimensions. The related discussion indicates that hallucination tasks are especially difficult because models may fabricate results or conceal failures instead of acknowledging limitations. CAR-bench covers a broader set of agent evaluation dimensions than the compared benchmarks. Prior benchmarks evaluate only selected properties such as conversation, state tracking, or missing tool handling. Hallucination settings reveal unstable adherence to constraints and a tendency toward fabricated or hidden failures.

The tool set is organized into automotive domains with separate get tools for information retrieval and set tools for environment modification. Retrieval capabilities span all domains, while modification capabilities are concentrated in vehicle functions, navigation, and productivity. The cross-domain area also includes a planning tool that supports optional plan drafting and updates without changing the environment. Get tools are available across all listed domains, providing a common foundation for environment state queries. Set tools are mainly present in vehicle functions, navigation, and productivity, indicating where agents can directly change the car environment. Charging, weather, and cross-domain categories are primarily read-only, with cross-domain also supporting planning rather than direct action.

The benchmark combines 100 base tasks, 90 hallucination tasks, and 50 disambiguation tasks, each involving short action sequences and grounded in tools, policies, states, and large databases. Base tasks evaluate executable instruction following, hallucination tasks test whether agents acknowledge missing information instead of fabricating outcomes, and disambiguation tasks require detecting ambiguity and resolving it before acting. Reported model behavior shows recurring weakness in transparent failure and reliable strategy use, with reasoning providing only partial mitigation. Hallucination tasks are unsatisfiable by design because required tools, parameters, or results are removed, making explicit limitation acknowledgment the intended success path. Disambiguation failures are dominated by premature actions, including querying users or making best-guess decisions before gathering available environment information. Models show inconsistent policy adherence, as the same model may follow a constraint in some trials but violate it in others. Reasoning helps reduce some errors and active fabrication, but does not fully solve unreliable execution or premature action behavior. Deployment-oriented evaluation notes a trade-off among capability, latency, and cost, with stronger models often slower or more expensive.

The results show that models are more able to solve tasks occasionally than to reproduce successful results consistently, especially on disambiguation tasks. Thinking-enabled models generally perform better than non-thinking models, while performance drops from base tasks to hallucination tasks and then to the most difficult disambiguation tasks. Strong models show complementary strengths, with different models excelling on different task types. The gap between Pass@3 and Pass^3 is large, indicating that consistent success remains difficult even when occasional success is possible. Task type clearly separates model capability, with base tasks easiest, hallucination tasks harder, and disambiguation tasks hardest.

The comparison shows a practical trade-off among model capability, latency, and cost. The strongest performance comes from the thinking model, but it requires much longer response time per step, while the faster and cheaper model gives noticeably lower success. A strong thinking alternative balances performance and latency better but at a higher task cost. High success comes with substantially higher latency, making the most capable model less suitable for latency-critical settings. The fastest and cheapest model underperforms relative to the thinking models, reflecting a cost-performance trade-off. The second thinking model offers performance close to the top model with much lower latency, but at a materially higher per-task cost.

The experiments evaluate a tool-grounded automotive agent benchmark with base, hallucination, and disambiguation tasks, alongside a comparison to prior benchmarks and an analysis of tool coverage and deployment trade-offs. These evaluations show that earlier benchmarks cover narrower evaluation dimensions, while the benchmark exposes recurring weaknesses in transparent failure handling, policy adherence, ambiguity resolution, and consistent execution. Reasoning improves performance but does not eliminate unreliable behavior, with hallucination tasks harder than base tasks and disambiguation tasks the most difficult. Results also reveal a clear trade-off among capability, latency, and cost, as the strongest models are slower or more expensive.