One-sentence Summary

Researchers from the University of Toronto, Stevens Institute of Technology, Dalhousie University, and the Vector Institute propose ACCORD, a framework and benchmark suite that introduces formal elements to disentangle commonsense grounding from reasoning in LLMs, enabling the automatic generation of multi-hop counterfactual benchmarks of arbitrary reasoning complexity that scale with model improvements.

Key Contributions

• ACCORD is introduced as a framework and benchmark suite that uses controlled, multi-hop counterfactuals to separate commonsense grounding from reasoning in large language models. It adds formal elements to control reasoning complexity, reasoning skills, and distractors beyond evaluations limited to one or two hops.

• ACCORD includes an anti-factual benchmark generator that produces arbitrarily many multi-hop instances from handwritten pairing templates. The generator uses tree duplication to reduce lexical matching bias and lets benchmark difficulty scale with future model improvements using minimal additional human effort.

• Experiments on ACCORD_CSQA, an application of ACCORD to CSQA, show state-of-the-art LLM performance degrading to random chance with moderate complexity scaling and reveal a significant gap between factual and anti-factual performance. A leaderboard and code for generating more complex benchmarks are released.

Introduction

LLMs are increasingly used for reasoning and as parametric knowledge bases, but their apparent reasoning performance can be confounded by memorized world knowledge, especially when contextual information conflicts with training priors. Prior commonsense reasoning benchmarks often rely on factual contexts, which can let models exploit parametric shortcuts, dataset artifacts, or inductive biases rather than genuinely reason from the given context. These benchmarks also typically lack the controlled complexity, explicit skills, and formal analysis found in logical or mathematical reasoning benchmarks. The authors introduce ACCORD, a framework for generating anti-factual commonsense reasoning benchmarks that use implausible contexts, disentangle contextual grounding from parametric reasoning, and scale reasoning complexity, skills, and distractors with minimal additional human effort.

Dataset

• The dataset described is ACCORD_CSQA, a benchmark suite built from the CommonsenseQA development set and ConceptNet. The authors use CSQA as the source of commonsense questions, concepts, and answer choices, and reuse manually defined reasoning skills from Talmor et al. to formalize how commonsense reasoning patterns can be compared.

• Dataset sources and composition:

  • Base examples come from CSQA, which is grounded in ConceptNet.
  • The authors assign each CSQA instance a ConceptNet relation by computing the majority vote among ConceptNet assertions that match an answer choice to the source concept.
  • Reasoning skills are converted into natural-language templates that connect two concepts through placeholders, such as an object-location or type-relation pattern.
  • These templates are composed into reasoning trees, which form the formal context used in the dataset.

• Balanced subset construction:

  • The inferred ConceptNet relation distribution is highly skewed, with some relations appearing hundreds of times and others only a few times.
  • To balance the benchmark, the authors randomly subsample each relation type down to 13 instances.
  • From the resulting 78 relation-balanced CSQA instances, 2 are removed because they involve inciting violence and 2 more are removed because valid pairing templates cannot be constructed.
  • The remaining 74 instances form the baseline ACCORD_CSQA^0 split.
  • Hand-crafted pairing templates expand these baseline instances to 93 pairing templates used to generate larger reasoning trees.

• Reasoning tree subsets:

  • The authors generate ACCORD_CSQA variants with reasoning tree sizes of 1 through 5.
  • The full generated suite contains 245,514 unique reasoning trees and guarantees at least 143 unique trees per pairing.
  • For efficient evaluation, they create a smaller benchmark by subsampling exactly one tree per unique combination of reasoning hops and distractors for each pairing.
  • This smaller benchmark contains 2,864 unique trees and also reduces the exponential weighting toward larger tree sizes.
  • Both the full suite and the smaller benchmark are released.

• Dataset artifact control and processing:

  • Because an answer choice could be guessed through lexical matching between the reasoning context and the answer options, the authors duplicate the paired tree once for each answer choice.
  • This keeps every answer choice present in the context while preserving the intended reasoning task.
  • Statement order is randomized within and between reasoning trees to reduce positional or recency bias.
  • Duplicate statements are removed to avoid biasing model reasoning toward repeated claims.
  • No image cropping or similar visual-data processing is described; the construction uses text-level balancing, subsampling, template creation, tree generation, statement removal, and randomization.

• How the data is used:

  • The paper uses ACCORD_CSQA as an evaluation benchmark rather than as a described training split.
  • ACCORD_CSQA^0 acts as a no-context baseline.
  • ACCORD_CSQA^1 through ACCORD_CSQA^5 provide increasing reasoning complexity by embedding answers in reasoning trees of different sizes.
  • The dataset is used to test whether large language models can answer commonsense questions when the formal reasoning context contains multiple plausible relations and answer-related statements.

Method

The authors leverage ACCORD as a systematic benchmark construction framework for generating anti-factual commonsense reasoning instances from an existing multiple-choice factual dataset. Given a question-answer instance with question $q$q and answer choices $\mathbf{a} = \{a_1, a_2, \ldots, a_Q\}$a={a1​,a2​,…,aQ​}, the framework augments the original item with an anti-factual context $C^{\mathrm{af}}$Caf that is aligned to a counter-world $\mathbf{w}^{\mathrm{af}} \neq \mathbf{w}^{\mathrm{def}}$waf=wdef. The goal is to construct the context so that the statements imply either the original factual answer $a^{\mathrm{def}}$adef or an alternative anti-factual answer $a^{\mathrm{af}}$aaf, while keeping the underlying reasoning structure as controlled as possible.

The overall pipeline is best understood through a concrete instantiation. Refer to the framework diagram below:

The first stage converts informal commonsense reasoning patterns into formalized templates. Each reasoning skill is represented as a recurring relational pattern over two concepts, such as “$X$X appears near $Y$Y.” These templates are derived from the reasoning structure needed to solve questions in the base dataset and serve as the building blocks for larger compositional structures. To make the templates relevant to a specific instance, the framework constructs pairing templates. A pairing template contains two variables, $\mathbf{V}_x$Vx​ and $\mathbf{V}_p$Vp​. The pairing variable $\mathbf{V}_p$Vp​ is grounded with a question-specific term $p$p, while the other variable remains free. The template is designed so that grounding the free variable to an answer choice, $\mathbf{V}_x \gets a_i$Vx​←ai​, distinguishes among the available answers. Each pairing template has a positive variant $p^+$p+ and a negative variant $p^-$p−, allowing it either to imply a given answer choice or to contradict it.

Generic reasoning structures are then built by composing multiple reasoning templates through shared variables. This produces a reasoning graph whose nodes are variables and whose edges are templates. To preserve valid reasoning behavior, the graph is constrained to be acyclic, so that composed templates cannot create circular entailments or paradoxical inference chains. Template composition is further restricted by reducibility: two templates may be linked only if their combination reduces to a valid commonsense reasoning pattern. This process prevents arbitrary template connections and ensures that each composed tree corresponds to coherent reasoning rather than an ad hoc sequence of statements.

Reasoning trees are generated automatically up to a maximum size hyperparameter $T$T. For each question, only those trees that match one of the question’s pairing templates are retained. These paired reasoning trees instantiate the compositional reasoning backbone of the anti-factual context. Within a paired tree, a reasoning path connects a source variable to a target variable, and the variable relevant to the answer choice is designated the answer variable $\mathbf{V}_a$Va​. Because the path structure is recursively reducible, grounding the answer variable to an answer choice, $\mathbf{V}_a \gets a_i$Va​←ai​, preserves the logical discrimination between answer choices established by the pairing template.

The framework quantifies reasoning difficulty by separating the relevant reasoning path from distractor templates. If a reasoning path contains $n$n templates and the full tree contains $T$T templates, then the number of distractor templates is

This formulation allows the authors to evaluate model robustness as the number of required inference steps and irrelevant statements vary. The key design property is that reasoning complexity can be increased systematically through tree growth, without requiring new manually annotated questions.

A major methodological concern is that an LLM might bypass formal reasoning by identifying the intended answer through superficial lexical matching between the context and one answer choice. To address this, the paired tree is duplicated once for each answer choice $a_i$ai​, ensuring that all choices appear in the anti-factual context. Statement order is randomized within and across duplicated trees, and duplicate statements are removed. These operations are intended to control artifacts such as recency bias or preference for repeated terms.

Variable grounding is performed through a backtracking beam tree search. The pairing variable and answer variable act as seed terms, and the remaining variables are filled anti-factually. ConceptNet is used as the grounding knowledge base, and triples that are not present in ConceptNet are treated as anti-factual. Because absence from a knowledge base does not strictly guarantee falsehood, the framework incorporates probabilistic hedging to reduce the risk of mistakenly labeling a valid factual relation as anti-factual.

The final stage selects which answer choice should be supported by logical implication. The pairing templates corresponding to all answer choices except one are negated, so that only the selected answer is implied while the others are contradicted. By choosing whether the positive variant is applied to the original factual answer or to another answer, the authors generate matched factual and anti-factual task variants. In both cases, the context contains anti-factually grounded statements, and all structural factors are held constant. This controlled construction enables direct comparison between performance on factual reasoning and performance on reasoning under an explicitly altered commonsense world.

Experiment

The experiments use an ACCORD benchmark derived from CSQA to evaluate large language models on increasingly complex reasoning trees under controlled settings. By comparing factual and anti-factual task variants, the results show that models perform much better when the correct answer aligns with common knowledge, revealing that apparent reasoning often reflects context unfaithfulness and reliance on memorized shortcuts. The study also disentangles the effects of reasoning hops and distractors, finding that multi-hop reasoning is the dominant source of degradation, while models are comparatively better at filtering distractors.

The mapping connects commonsense reasoning skills, such as spatial, causal, and part-whole patterns, to ConceptNet relations and named reasoning templates. These templates express recurring patterns in simple form, such as X appears near Y, and can be composed into reasoning trees for the evaluation framework. The accompanying results indicate that factual reasoning tasks are easier than anti-factual variants, suggesting that parametric familiarity can confound measured reasoning ability. Each reasoning skill is represented by a short template linking two concepts through a named pattern. Performance degrades more sharply with increasing reasoning hops than with increasing distractors when other factors are controlled. The gap between factual and anti-factual results points to context unfaithfulness and reduced construct validity in factual commonsense benchmarks.

ACCORD is described as combining counterfactuals, anti-factual grounding, explicit reasoning skills, composability, automatic scaling, and measurable reasoning hops and distractors. Related commonsense reasoning benchmarks are shown to provide only parts of this feature set, often lacking explicit skills, composability, scalability, measurable complexity, or anti-factual counterfactuals. ACCORD is the only listed benchmark marked as having counterfactuals, anti-factuals, explicit skills, composability, and scalability. ACCORD also provides measurable reasoning hops and distractors, while several related benchmarks report unknown or unmeasured complexity. Prior benchmarks vary in counterfactual support, with some offering anti-factual grounding but not composability or controlled skills.

The evaluation maps commonsense reasoning skills, including spatial, causal, and part-whole patterns, to ConceptNet relations and compositional reasoning templates, and compares ACCORD with existing benchmarks. Experiments show that factual tasks are generally easier than anti-factual variants, indicating that parametric familiarity can confound measured reasoning and weaken construct validity, while increasing reasoning hops affects performance more strongly than adding distractors. Overall, ACCORD is positioned as a benchmark that uniquely combines counterfactual and anti-factual grounding, explicit skills, composability, scalable generation, and measurable complexity, whereas prior datasets provide only partial support for these features.