One-sentence Summary

Researchers from the Chinese Academy of Sciences and Beijing Academy of Artificial Intelligence introduce RuleTrek, a multi-domain benchmark for agentic rule reasoning whose rule-grounded pipeline extracts structured rule graphs and composes consistent complex sub-scenarios; on 11 frontier LLMs, Rule Identification and Rule Deduction reach only 62.47% and 60.43%, respectively, while End-to-End Rule Reasoning scores range from 26.97% to 55.29%.

Key Contributions

• RuleTrek is introduced as a multi-domain benchmark for agentic rule reasoning in complex, multi-file scenarios, evaluating Rule Identification, Rule Deduction, and End-to-End Rule Reasoning in interactive workspaces.
• A rule-grounded scenario construction pipeline extracts structured rule graphs, generates rule-specific sub-scenarios, and incrementally composes them while ensuring scenario consistency and rule-based solvability.
• Experiments on 11 frontier LLMs show the highest Rule Identification and Rule Deduction scores are 62.47% and 60.43%, while End-to-End scores range from 26.97% to 55.29%, indicating substantial challenges in identifying and applying rules over distributed information.

Introduction

Recent advances in large language models have enabled agents to handle complex domain-specific tasks such as tax assessment, where solving a problem requires identifying applicable rules and integrating distributed scenario information across multiple files. Existing rule-reasoning benchmarks tend to use simplified scenarios, while realistic workspace benchmarks rarely evaluate whether agents can identify and correctly apply domain-specific rules in such complex settings. The authors address this gap by introducing RuleTrek, a multi-domain benchmark with interactive workspaces and a rule-grounded scenario construction pipeline that evaluates rule identification, rule deduction, and end-to-end rule reasoning.

Dataset

The authors describe RuleTrek as a benchmark dataset built from real-world rule repositories.

Sources and Domains

• RuleTrek covers six domains:
  • Taxation
  • PostgreSQL
  • Cybersecurity
  • Software policy
  • Procurement
  • Game rules
• For each domain, the authors collect relevant real-world rule files and preserve their original organization and formats, including documents, tables, configuration files, and source code.
• The excerpt does not report the exact number of instances or per-domain sizes.

Instance Composition and Schema

Each benchmark instance contains four components:

• Rule repository $\mathcal{R}_d$Rd​
• Verified rule graph $\Gamma_d$Γd​
• Scenario $S_T$ST​
• Question $Q$Q

Dataset Construction and Processing

RuleTrek uses a three-stage pipeline. All LLM-based construction steps use GPT-6-Astra.

• Stage 1: Domain Rule Graph Extraction

  • An LLM extracts a structured rule graph from the collected rule files.
  • Nodes represent individual rules grounded in the original repository.
  • Edges capture relationships such as applicability dependencies, eligibility conditions, aggregation, exceptions, overrides, precedence, and temporal constraints.
  • Human experts manually verify the extracted rules, source references, and dependencies, revising inaccurate or missing information.

• Stage 2: Rule-Specific Sub-Scenario Construction

  • For each rule, an LLM generates a sub-scenario that requires that specific rule to resolve.
  • Sub-scenarios use domain-appropriate files such as spreadsheets, configuration files, logs, forms, messages, tables, or source code.
  • Experts manually review each sub-scenario for rule consistency, solvability, and sufficient information.
  • Redundant cases are removed, and experts ensure diversity in entities, contexts, and information organization.

• Stage 3: Dependency-Aware Composition

  • Sub-scenarios are incrementally composed according to dependencies in the verified rule graph.
  • When a new sub-scenario is added, existing files may be revised to align entities, timestamps, states, versions, units, and cross-file references.
  • Human experts check consistency and solvability after each composition step.
  • If a check fails, the LLM revises the affected files and the result is re-verified.
  • The original rule graph remains unchanged during composition.

• Question Instantiation

  • After the final scenario passes consistency and solvability checks, the authors generate a question and its answer based on the scenario and rule graph.
  • Human experts solve each question using the original rules and scenario files, verifying answerability, answer uniqueness, and rule compliance.
  • Invalid instances are revised and revalidated.

Filtering and Quality Control

• Manual verification is applied to rule graphs, sub-scenarios, intermediate composed scenarios, and final questions.
• Consistency checks examine whether information remains coherent across files.
• Solvability checks determine whether the scenario can be resolved using the underlying rules.
• Experts also verify answer uniqueness and rule compliance.

How the Data Is Used

RuleTrek is presented as an evaluation benchmark rather than a training corpus. The excerpt does not describe training splits or mixture ratios.

It defines three evaluation settings over the same complex scenarios and questions:

• Rule Identification (RI)

  • Input: rule repository $\mathcal{R}_d$Rd​, scenario $S_T$ST​, question $Q$Q
  • Goal: identify the applicable rules and their dependencies
  • Output: an estimated rule graph $\hat{\Gamma}$Γ^
  • The verified rule graph and final answer are not required.

• Rule Deduction (RD)

  • Input: verified rule graph $\Gamma_d$Γd​, scenario $S_T$ST​, question $Q$Q
  • Goal: apply the provided rules to the scenario and derive the final answer
  • Output: final answer $\hat{A}$A^

• End-to-End Rule Reasoning (E2E)

  • Input: rule repository $\mathcal{R}_d$Rd​, scenario $S_T$ST​, question $Q$Q
  • Goal: identify applicable rules from original documents, resolve dependencies, and apply them to produce the final answer
  • Output: final answer $\hat{A}$A^

Agents in all settings work in interactive workspaces and may search and inspect files and execute code as needed. No image cropping or vision-specific preprocessing is described; the data remains in domain-specific file formats.

Method

The authors develop a rule-grounded scenario construction pipeline called RuleTrek, which builds complex scenarios grounded in real-world rule repositories. The pipeline consists of three primary stages followed by a question instantiation phase. All LLM-based construction steps utilize GPT-6-Astra.

In the first stage, Domain Rule Graph Extraction, the authors collect a set of relevant real-world rule files $\mathcal{R}_d$Rd​ from six diverse domains, including taxation, PostgreSQL, cybersecurity, software policy, procurement, and game rules. An LLM is employed to extract a structured rule graph $\Gamma_d = (V_d, E_d)$Γd​=(Vd​,Ed​) from these repositories. Each node $R_i \in V_d$Ri​∈Vd​ represents a rule grounded in the original repository, while edges in $E_d$Ed​ capture explicit relationships such as applicability dependencies, eligibility conditions, aggregation, exceptions, overrides, precedence, and temporal constraints. Human experts then manually verify the extracted graph against the original repository to ensure the correctness of rules, source references, and dependencies, revising any inaccuracies.

The second stage focuses on Rule-Specific Sub-Scenario Construction. To maintain control over which rules are exercised and ensure scenario consistency, the authors avoid directly generating complex scenarios from the rule graph. Instead, for each rule $R_i \in V_d$Ri​∈Vd​, an LLM generates a corresponding sub-scenario $s_i$si​ based on $R_i$Ri​ and its dependencies in $\Gamma_d$Γd​. Each sub-scenario provides the necessary information to apply $R_i$Ri​ and is represented through domain-specific files such as spreadsheets, configuration files, logs, forms, or source code. These files supply the entities, events, and temporal or state information required for rule application. Human experts manually review each sub-scenario for rule consistency, solvability, and sufficient information, while also ensuring diversity in entities and contexts.

In the third stage, Dependency-Aware Composition, the authors use an LLM to incrementally compose the generated sub-scenarios according to the dependencies encoded in $\Gamma_d$Γd​. Starting from an initial scenario $S_1$S1​, additional sub-scenarios are systematically integrated to produce the final complex scenario $S_T$ST​. The composition operation at each step is defined as:

where $s_{i_{t+1}}$sit+1​​ is the newly introduced sub-scenario and $\mathcal{C}_{\Gamma_d}$CΓd​​ denotes LLM-assisted composition guided by rule dependencies. This process involves more than simply combining files; introducing a new sub-scenario may require revising existing files to align entities, timestamps, states, versions, units, and cross-file references. After each composition step, human experts verify the consistency and solvability of the intermediate scenario. If either check fails, the LLM revises the affected files, followed by further human verification. Composition proceeds only when both checks are satisfied, resulting in a final scenario $S_T$ST​ where interacting rules and information are distributed across multiple files.

Finally, in the Question Instantiation phase, once the final scenario $S_T$ST​ passes all checks, the authors generate a question $Q$Q and its answer $A$A based on $S_T$ST​ and $\Gamma_d$Γd​. Human experts solve each question using the original rules and scenario files to verify answerability, answer uniqueness, and rule compliance. Invalid instances are revised and revalidated, ensuring each benchmark instance comprises the rule repository $\mathcal{R}_d$Rd​, the verified graph $\Gamma_d$Γd​, the scenario $S_T$ST​, and the question $Q$Q.

Experiment

RuleTrek evaluates eleven frontier LLMs on 240 complex scenarios across six domains in interactive workspaces with file and code access, using three settings: rule identification, rule deduction with verified rule graphs, and end-to-end rule reasoning. The experiments validate that agents struggle both to identify applicable rules and dependencies from original documents and to correctly apply provided rules, with rule-selection errors being the most common failure. Performance further declines as rule-graph size and depth increase, and model behavior differs in interaction efficiency, while automatic scoring shows substantial agreement with human judgments and stable rankings across judge families.

Prior rule-reasoning benchmarks vary in whether they require applying domain rules to concrete cases, but none provide distributed scenario evidence or interactive workspace support. Most also retain native rules only partially or not at all, with full native rule support absent across the listed benchmarks. This leaves a gap for evaluating rule identification and application in complex multi-file settings. RuleArena and RuleWeaver fully support rule application and multi-domain coverage, but they still lack distributed scenarios and interactive workspaces. No listed rule-reasoning benchmark supports distributed evidence or workspace-based operation, and native rule representations are at most partially preserved.

Frontier LLMs exhibit limited rule reasoning across all tested domains and tasks. Rule identification is particularly difficult, with the best average identification score only near 62% and rule-selection errors dominating failure trajectories. Even when verified rules are supplied, deduction peaks around 60%, and end-to-end performance remains lower, indicating that both identifying and applying complex rules constrain overall accuracy. Identifying applicable rules is a major bottleneck: the best average rule identification score is near 62%, while the lowest model is around 33%. Even when verified rules are provided, deduction peaks near 60%, and end-to-end scores span roughly 27% to 55%, showing that rule application also remains difficult.

Error analysis shows models frequently miss condition-specific controls in cybersecurity, supporting data and initialization rules in tax, and distinct movement rules for different travel paces in game scenarios. These errors reflect wider rule-selection failures, the largest source of agent failures, even when relevant evidence is available. Complex scenarios make it hard for agents to identify complete and correctly structured rule sets. Models identify general cybersecurity requirements but overlook fine-grained firewall and remote-access controls. Tax scenarios miss necessary supporting rules for data loading and tax-year initialization. Game scenarios incorrectly merge movement rules that apply to different travel paces under distinct conditions. Rule-selection errors are the most common failure category, indicating persistent difficulty in identifying applicable rules.

Automatic scoring reliability is strong across retrieval, reasoning, and end-to-end tasks, with human-judge agreement consistently high but slightly below human-human agreement. Cross-family rescoring with alternative judge models produces robust submission-level agreement and highly stable model rankings. Automatic judge scores agree substantially with independent human annotations across all three tasks. Human-human agreement remains higher than human-judge agreement. Scoring behavior and model rankings remain stable when submissions are rescored by judges from different model families.

The evaluation first reviews existing rule-reasoning benchmarks and finds that none combine distributed scenario evidence, interactive workspaces, and full native rule support, with RuleArena and RuleWeaver covering rule application and multiple domains but not these settings. Frontier LLM experiments across those domains show limited rule reasoning: rule identification is the main bottleneck, with average identification scores around 62% and rule-selection errors dominating, while deduction peaks near 60% and end-to-end accuracy remains lower. Error analysis attributes these failures to missed condition-specific controls, supporting rules, and distinct movement rules in complex cybersecurity, tax, and game scenarios. Automatic scoring is validated as reliable, with human-judge agreement substantially high across retrieval, reasoning, and end-to-end tasks, and cross-family rescoring yields stable rankings.