One-sentence Summary

Researchers from Beihang University, Renmin University, et al. propose REALCLI, a paired benchmark unifying CLI task execution and human-agent collaboration evaluation through a five-dimensional framework with 22 sub-dimensions and 51 scenario templates, where REALCLI-EXEC supplies full specifications upfront and REALCLI-COLLAB introduces a scripted user delivering progressive information, corrections, constraint revisions, and authorization decisions, revealing that frontier agents achieve only 49.2% average score and 37.6% on the joint collaboration score $J_{\mathrm{collab}}$Jcollab​.

Key Contributions

• REALCLI is introduced as a paired benchmark that unifies evaluation of task execution and human-agent collaboration on 101 realistic CLI tasks spanning 10 categories and 52 subdomains, with REALCLI-EXEC and REALCLI-COLLAB modes sharing task goal, environment, and verifier.
• REALCLI defines a five-dimensional collaboration framework operationalized into 22 sub-dimensions and 51 interaction scenario templates, covering understandability, steerability, observation and recovery, trust calibration, and safety and risk awareness, and scores collaboration behavior from interaction traces and environment evidence.
• An evaluation across 33 configurations combining 11 frontier models and three agent harnesses shows REALCLI remains challenging, with a peak joint collaboration score of 37.6%, and reveals that task effectiveness diverges from collaboration quality, with consistent weaknesses in steerability, trust calibration, and safety boundary management.

Introduction

Large language models are enabling coding agents to act through command-line interfaces, with benchmarks like SWE-bench and Terminal-Bench evaluating autonomous execution on repository and terminal tasks. However, these protocols typically provide complete and fixed specifications, judge only the final state, and offer limited assessment of the execution process or user coordination. Prior work that includes user interaction usually treats it as a means to task completion or focuses on narrow behaviors such as instruction following, rather than evaluating collaboration quality itself. To address this gap, the authors build a five-dimensional collaboration framework grounded in HCI theory and introduce REALCLI, a paired benchmark of 101 realistic CLI tasks that runs the same tasks in autonomous and scripted human-agent collaboration modes, enabling direct comparison of task outcomes and collaboration quality.

Dataset

Dataset Description: REALCLI

The authors construct REALCLI as a benchmark that pairs autonomous execution with user collaboration on the same CLI tasks and environments. Both modes share task goals and acceptance criteria, while COLLAB adds reproducible scripted interactions and evaluates collaboration quality alongside task outcomes.

Composition and schema

• Each task is formalized as a tuple: $x = (\mathcal{E}, s_0, \omega, V)$x=(E,s0​,ω,V).
  • $\mathcal{E}$E: CLI execution environment
  • $s_0$s0​: initial environment state
  • $\omega$ω: complete task specification
  • $V$V: outcome verifier
• REALCLI evaluates each task under two modes:
  • EXEC: autonomous execution using only environment actions and observations
  • COLLAB: execution with environment actions plus communication actions, and environment observations plus user messages
• The final dataset includes:
  • 101 tasks
  • 10 top-level categories
  • 52 subdomains
  • 51 collaboration scenario templates
  • 1,658 evaluation checklist items

Sources and construction subsets

• Base instances are synthesized from real CLI logs.
• There are two main construction-stage subsets:
  • Validated base-instance pool
  • Final interactive delivery pool

Base instance synthesis

• Real CLI logs are compressed into task goals, environment states, constraints, and failure signals.
• Skill-oriented instance construction converts this evidence into reusable skills covering operational patterns, environment dependencies, and corrective actions.
• These skills guide synthesis of task instructions, containerized environments, reference solutions, and verifiers.
• Quality checks inspect files and container configurations, verify specification–test alignment, and require reference solutions to pass while no-op baselines fail.
• Multi-agent rollouts distinguish task issues from infrastructure issues. Task issues are routed to instance repair; infrastructure issues are retried.
• Difficulty profiling classifies candidates as easy, medium, or hard.
  • Medium and hard candidates enter the validated base-instance pool.
  • Easy candidates are augmented with additional skills and re-enter instance construction and validation.

Interactive instance synthesis

Each validated base instance is augmented with an interaction layer supporting gradual information disclosure and scenario-driven probes.

• Scenario assignment: each base task receives six collaboration scenarios based on task semantics, execution characteristics, and predefined templates.
• Interactive script authoring: scripts include an opening message, information slices, scenario probes, and checklist items. Opening messages give partial information, while later slices disclose remaining requirements and context.
• Script validation: static checks verify structure, file references, and alignment among scenarios, probes, and checklist items. Semantic review checks task–scenario fit, dialogue naturalness, and evidence traceability.
• End-to-end pilot: a target agent completes the task with a user simulator, and judge review confirms that each intended collaboration capability is elicited and measurable.
• Final curation: human experts review candidates and apply difficulty filtering. Passing instances retain base task artifacts and add interaction-specific files before entering the delivery pool.

Usage in evaluation

• In EXEC, the initial observation presents the complete task specification, after which observations come only from the environment.
• In COLLAB, the run begins with a partial request, and the remaining task information is progressively introduced through user messages. The agent may also request clarification or authorization.
• Both modes use the same outcome verifier on the terminal state.
• COLLAB additionally uses the full interaction trace to assess collaboration quality.
• Collaboration quality is measured along five dimensions, with sub-dimensions refining specific behaviors and scenario templates grounding them in concrete CLI task situations.
• The provided material does not specify a training split or mixture ratios; REALCLI is presented as an evaluation benchmark rather than a training corpus.

Method

The authors construct REALCLI through two sequential pipelines: base instance synthesis from real CLI logs and interactive instance synthesis from validated base instances, as shown in the figure below:

Base Instance Synthesis. The process begins with Log Compression, which condenses raw CLI traces into task goals, environment states, constraints, and failure signals. During Skill-Oriented Instance Construction, this evidence is transformed into reusable skills capturing operational patterns, environment dependencies, and corrective actions. These skills guide the synthesis of task instructions, containerized environments, reference solutions, and verifiers. Quality Checks then inspect files and container configurations, specification-test alignment, and verify that reference solutions pass while no-op baselines fail. Candidates undergo multi-agent, multirun rollouts. Agent Rollout & Diagnosis distinguishes task issues from infrastructure issues, routing task issues back to instance repair while retrying infrastructure issues. Finally, Difficulty Profiling classifies the resulting candidates as easy, medium, or hard. Medium and hard candidates enter the validated base-instance pool, whereas easy candidates are augmented with additional skills and re-enter instance construction and validation.

Interactive Instance Synthesis. Each validated base instance is augmented with an interaction layer supporting gradual information disclosure and scenario-driven probes across five stages. First, in Scenario Assignment, six distinct collaboration scenarios are assigned to each base task based on task semantics, execution characteristics, and predefined templates, balancing coverage across task types and the five-dimensional collaboration framework. Next, Interactive Script Authoring produces an interaction script comprising an opening message, information slices, scenario probes, and checklist items. The opening message provides partial information, while information slices progressively disclose requirements and context. Each probe specifies a target collaboration behavior, expected evidence, and linked checklist items. Probes are rendered as user messages presented to an uninformed agent, with revisions made if they prematurely leak evaluation intent. Script Validation follows, where Static Checks verify script structure, file references, and alignment among scenarios, probes, and checklist items, while Semantic Review assesses task-scenario fit, dialogue naturalness, and evidence traceability. Failed scripts return to authoring. An End-to-End Pilot is then conducted where a target agent completes the task alongside a user simulator, followed by a Judge Review to confirm that each intended collaboration capability is successfully elicited and measurable. Finally, candidates passing the pilot undergo Final Curation through human expert review and difficulty filtering. Instances that pass retain their base task artifacts, append specific files required for user interaction, and enter the delivery pool.

Script-Driven User Simulation. REALCLI uses a task-level user script to control how task information enters the interaction and when the user participates. Its core consists of information slices and interaction probes. For task $x$x, the simulator at the $k$k-th user interaction is defined as:

where $c_{x}$cx​ specifies the user role and the opening and closing of the interaction, $\mathcal{I}_{x}$Ix​ and $\mathcal{P}_{x}$Px​ denote the information slices and interaction probes, and $z_{k}$zk​ denotes the current runtime state. Information slices $\mathcal{I}_{x}$Ix​ decompose the complete task specification $\omega$ω into slices released progressively during execution, either in response to agent inquiries or proactively supplied by the simulator. Interaction probes $\mathcal{P}_{x}$Px​ instantiate the scenario templates of the five-dimensional collaboration framework into probes tied to the current task. Each probe contains a scripted user intervention with its trigger condition, expected behavior, and evidence requirements. At the $k$k-th interaction, the simulator selects the applicable information slices and probes based on the preceding execution trajectory and state $z_{k}$zk​, generates the user message, and updates the state.

Experiment

REALCLI evaluates CLI agents on the same 101 tasks under autonomous execution and script-driven user collaboration, measuring task success alongside a five-dimensional collaboration quality framework. The experiments show that current frontier agents optimize for task completion but remain weak in steerability, trust calibration, and safety, so joint collaboration scores drop sharply even when task success is moderate. User interaction produces mixed outcomes, helping models that ask for clarification and integrate feedback while hurting those that act on partial information, and the clearest weakness is maintaining goals and constraints across turns. Many trajectories that complete the task still fail collaboration checks or produce artifacts that mismatch negotiated constraints, indicating that execution success and collaboration quality are distinct and complementary evaluation axes.

The benchmark landscape shows broad environment-level verification but sparse explicit evaluation of runtime user participation, paired evaluation, and collaboration dimensions such as understandability, steerability, observation, trust, and safety. Experimental results indicate models are more reliable at responding to immediate blockers and successive requests than at maintaining prior constraints across turns. Even successful interactive trajectories often miss collaboration requirements, and final artifacts can mismatch stated constraints despite appropriate dialogue responses. Environment-level verification is explicitly evaluated across the listed benchmarks, while paired evaluation and most collaboration dimensions are largely marked as not evaluated or only partially evaluated. Models respond more often to observed blockers and successive requirement changes than to goal expansion, conflicting instructions, or retaining earlier constraints, pointing to weak cross-turn goal maintenance.

The collaboration framework defines five dimensions: understandability, steerability, observation and recovery, trust calibration, and safety and risk awareness. Subdimensions refine specific behaviors within each dimension, and scenario templates ground them in concrete CLI task situations. Experimental evidence indicates that models respond more reliably to immediate requests than to cross-turn goal maintenance, constraint retention, or alignment between dialogue and final artifacts. Models handle immediate blockers and current requests more reliably than scope expansion, conflicting instructions, or retaining earlier constraints. Successful task completion does not guarantee successful collaboration; many trajectories still fail at least one applicable collaboration check, especially around authorization, conflict handling, or risk communication.

The results show a consistent gap between autonomous execution and joint collaboration across all three scaffolds: execution rewards are moderate to high, but joint collaboration scores fall sharply when safety, communication, and coordination constraints are enforced. Claude Opus 5 leads the held-out split, followed by DeepSeek V4.1 Flash. User interaction changes outcomes, with clarification-seeking models improving collaboration rewards while many frontier models still fail collaboration checks. Joint collaboration scores are consistently lower than execution rewards across models and scaffolds, and some models lose a large share of their execution score under joint collaboration evaluation. Models that actively ask for clarification and incorporate user corrections show positive collaboration reward changes, while overconfident execution from partial information is associated with negative changes.

Collaboration capabilities are uneven across models and scaffolds, with relative strength in Understandability and Observation & Recovery but consistent weakness in Steerability, Trust Calibration, and Safety Boundary Management. Joint collaboration scores are always below the corresponding composite collaboration scores, showing that stricter collaboration requirements reduce performance. The strongest model remains relatively stable across scaffolds, while others drop sharply under joint evaluation. Understandability and Observation & Recovery scores tend to be higher, whereas Steerability, Trust Calibration, and Safety Boundary Management are the weakest dimensions. Joint collaboration scores fall below composite collaboration scores in all reported model-scaffold cases, and some models reach near-zero Safety Boundary Management scores.

The experiments evaluate CLI task benchmarks through a five-dimension collaboration framework that contrasts environment-level execution success with joint collaboration checks across different model scaffolds. They show that models respond more reliably to immediate blockers and successive requests than to cross-turn goal maintenance, constraint retention, or alignment between dialogue and final artifacts, and successful task completion often still misses collaboration requirements such as authorization, conflict handling, and risk communication. Across models and scaffolds, joint collaboration scores fall consistently below execution and composite collaboration scores, with relative strengths in Understandability and Observation & Recovery and persistent weaknesses in Steerability, Trust Calibration, and Safety Boundary Management, while clarification-seeking behavior improves collaboration outcomes.