One-sentence Summary

Qilu University of Technology and Southeast University introduce CHECKWHY, a 19K-instance causal fact verification dataset that requires models to verify causal relations through argument structures progressing from foundational evidence to claim establishment, revealing that fine-tuned models and CoT-prompted LLMs struggle to generate satisfying argument structures.

Key Contributions

• The paper introduces CHECKWHY, a human and model collaborative dataset and benchmark for causal fact verification of “why” claims, containing over 19K claim-evidence-argument triplets labeled supports, refutes, and not enough info.
• The work constructs argument structures with convergent, linked, and divergent patterns to explicitly represent the reasoning process from foundational evidence to the causal conclusion within each claim.
• Experiments on state-of-the-art models, with automated and human evaluation, show that argument structures are important for causal verification and that fine-tuned models or chain of thought prompted LLMs struggle to produce satisfactory argument structures.

Introduction

Fact verification is important because it helps check online claims against reliable evidence and reduces the spread of misinformation. Prior benchmarks have largely focused on simple factoids such as who, what, when, and where, where verification can often be done through direct evidence matching, so they provide limited insight into explicit causal or multi-step reasoning. The authors address this gap with CHECKWHY, a benchmark for verifying why claims that contain causal relations. They construct 19,596 claim, evidence, and argument structure triplets using human model collaboration and inspiration from logical theory, and evaluate fine-tuned models and LLMs to show that explicit argument structures are valuable but difficult to generate.

Dataset

• Dataset source: The authors use WIKIWHY, a public cause and effect QA dataset, as the starting source. They begin from 9,406 WIKIWHY cause-effect instances.

• Initial filtering: They remove pairs with incomplete syntactic structure in either the cause or effect by checking for a missing predicate with Stanford NLP parsing. This leaves 7,403 valid cause-effect pairs for further construction.

• Overall construction: The authors build CHECKWHY through a human-model collaboration pipeline: data preparation, GPT-4 generation, and manual validation.

• Label distribution: For each cause-effect pair, GPT-4 generates three instance types with balanced veracity labels: SUP for support, REF for refute, and NEI for not enough info.

• SUP evidence generation: GPT-4 receives the cause-effect pair and explanation from WIKIWHY, then produces stepwise reasoning evidence that supports the causal claim.

• REF evidence generation: The authors use a counterfactual strategy. GPT-4 first generates an opposite or counterfactual effect, then reasons from the original cause to that counterfactual effect to produce evidence that refutes the original claim.

• NEI evidence generation: NEI instances are created by removing evidence from SUP or REF instances or by merging evidence from the two, making the claim unverifiable from the provided evidence alone.

• Argument structures: GPT-4 also generates structured reasoning for SUP and REF instances using Python-style examples. The schema uses FACT nodes for evidence and function-style links for series, divergent, and combined reasoning paths.

• Claim generation: GPT-4 turns each cause-effect pair into varied natural-language causal claims without external information access.

• Distractor evidence: To simulate misleading verification settings, BM25 retrieves 300 highly lexical-overlapping evidence candidates, and BLEURT selects 8 to 12 semantically similar distractors for each instance.

• Human editing: Annotators review and edit claims, evidence, and argument structures. They check that claims faithfully cover the cause-effect pair, evidence matches the label, and argument structures are concise, grammatical, and logically valid.

• Manual filtering: A second group filters low-quality edited instances using three criteria: faithfulness, fluency, and validity. After filtering, 6,757 SUP and 6,638 REF instances are retained.

• Quality control: Two annotators edit and filter each instance, and a third resolves disagreements. The authors report Fleiss k of 0.75 for final retained instances.

• Final scale: CHECKWHY contains 6,532 complete instance sets, each with SUP, REF, and NEI instances, for a total of 19,596 instances.

• Instance schema: SUP and REF instances include claim, evidence, and argument structure. NEI instances include only claim and evidence because they lack a valid argument structure.

• Reasoning subset: Because NEI does not include argument structure, the authors also build a CHECKWHY variant excluding NEI to evaluate reasoning ability through SUP and REF argument structures.

• Dataset statistics: The dataset averages 13.7 evidence pieces and 4.08 reasoning steps per instance.

• Reasoning categories: A sample analysis identifies event causality at 36%, inference from properties at 22%, rule-based inference at 17%, inductive reasoning at 14%, and sequential reasoning at 10%.

• Usage: The authors use CHECKWHY as a fact verification dataset with balanced support, refute, and not enough info labels, and as a benchmark for argument-structure reasoning. The provided dataset section describes construction and evaluation use, not explicit training splits or mixture ratios.

Method

The authors leverage a human and model collaborative pipeline to construct the CHECKWHY annotation framework. As shown in the figure below:

The pipeline consists of data preparation, generation with large language models, and manual validation. The first stage parses source causal records to extract cause and effect units, which become the basic inputs for claim generation, evidence generation, and argument structure modeling.

Before generation, the framework represents verification evidence as a directed argument tree. The claim serves as the root node, and evidence nodes appear as child nodes connected by edges that encode logical support relations. The design merges related reasoning patterns into a small set of structural operators. A series link moves from one evidence node to one parent node. A divergent link moves from one evidence node to multiple parent nodes. A combined link moves from multiple evidence nodes to one parent node. To produce both supporting and refuting instances for the same causal claim, the framework applies different structures to the same claim. One structure establishes the causal relation, while another structure encodes reasoning that opposes the claimed causal relation.

For model assisted generation, the authors use GPT-4 with in-context learning. For each cause and effect pair, the process is divided into several modules. The support evidence module inputs the cause, the effect, and the source explanation, then uses ReAct style prompts to encourage step by step reasoning from cause to effect. The generated reasoning chain is treated as support evidence.

The refutation module uses a counterfactual construction strategy. Starting from the original effect, the model is prompted to produce a counterfactual effect with an opposite label. It is then prompted to reason from the original cause to this counterfactual effect. The resulting evidence is interpreted as refuting the original effect. This two step design narrows the generation space for difficult refutation cases.

The NEI evidence module creates instances with not enough info labels. It can remove evidence from a supporting or refuting instance, or merge evidence from both types, so that the remaining evidence is insufficient to verify the claim with confidence.

For each generated instance, the framework also produces a compact argument structure. GPT-4 is prompted with Python style code templates that impose strict formatting. The templates introduce a FACT class for evidence nodes and three linking functions corresponding to the series, divergent, and combined argument patterns. These functions guide the model to output an executable style reasoning path rather than free form prose.

The claim generation module inputs the cause and effect pair and prompts the model to express the causal claim in diverse natural language forms without accessing external information. To simulate misleading verification scenarios, the pipeline also constructs distractor evidence. BM25 retrieves candidates with high lexical overlap with the claim, and BLEURT selects a smaller set of candidates with close semantic similarity. These selected distractors are attached to the SUP, REF, and NEI instances.

The final stage performs human validation in two modes. In editing mode, annotators with NLP backgrounds review and refine the claim, evidence, and argument structure. They check that the claim preserves the original causal details, that evidence aligns with its label, that the language is concise, and that the argument structure removes redundant non code content. In filtering mode, separate annotators decide whether to retain or discard edited instances according to faithfulness, fluency, and validity. Faithfulness requires the evidence to verify the claim under the assigned label. Fluency requires grammatical and readable expressions. Validity requires the argument structure to correctly depict the reasoning from cause to effect.

Quality control is applied through redundant annotation. Two annotators perform editing and filtering for each instance. A third annotator resolves disagreements, and majority voting determines the final retained instance.

Experiment

The experiments evaluate CHECKWHY through four increasingly difficult tasks: verifying claims from leaf evidence, incorporating given argument structures, jointly predicting labels and argument structures from evidence, and constructing complete reasoning chains from leaf evidence alone, using discriminative, generative, and LLM baselines. Automatic and human evaluations show that providing explicit argument structures consistently improves verification, while ambiguous not-enough-information evidence remains especially challenging for LLMs. Overall, explicit structural reasoning is central to the benchmark, CodeT5-style models perform best on structure generation, and LLMs remain limited in constructing reliable causal argument structures, indicating substantial room for improvement.

The table summarizes the CHECKWHY corpus and its SUP/REF subset used for argument-structure reasoning. The full set contains more instances, while the subset focuses on supporting and refuting claims and shows higher average evidence, including distractors. Claims remain similarly short across both versions, indicating that extended evidence rather than longer claims characterizes the data. The full CHECKWHY set is larger and balanced across train, dev, and test splits. The SUP/REF subset has higher average evidence and includes distractor evidence, reflecting detailed reasoning inputs. Average claim length stays nearly identical across the full set and the subset. Relative to WIKIWHY, CHECKWHY is described as having more extended reasoning evidence and steps.

The sampled argument structures show several recurring forms of reasoning, with event causality appearing most prominently. Inference from properties and rule-based inference also account for important portions of the examples, indicating that reasoning often relies on explicit causal links or structured conditions. Event causality is the most prevalent reasoning type in the sampled instances. Property-based and rule-based reasoning are also common patterns in the argument structures.

Fine-tuned discriminative models generally outperform the generative CodeT5 baseline on causal verification, especially when argument structures are provided. Moving from text-only Task 1 settings to structure-aware Task 2 yields consistent gains across baselines, with Transformer-XH showing the strongest benefit. Ambiguous evidence settings produce mixed effects, improving most baselines but slightly lowering CodeT5 performance. Discriminative models achieve the strongest verification accuracy and F1 scores overall. Transformer-XH benefits most from argument structures among the baselines. CodeT5 trails discriminative models and does not improve when moving from Y3 to Y2.

Across the argument generation setting, CodeT5+ is the strongest baseline on automatic metrics, outperforming both CodeT5 and LLMs on structure similarity, exact match, accuracy, and F1. LLMs still lag behind fine-tuned models automatically, although GPT-4 can obtain more favorable human comparisons, suggesting that fluency and semantic correctness do not translate into overall superiority. Human evaluation indicates that all baselines remain limited, leaving substantial room for improvement. CodeT5+ leads automatic argument generation and verification metrics. LLMs are weaker on automatic scores but can be more competitive in human judgments. Structure similarity is consistently higher than exact match, reflecting semantic overlap rather than verbatim matching. Even the strongest baseline shows clear room for improvement, with only modest human-rated validity.

Experiments analyze the CHECKWHY corpus and its SUP/REF subset, sampled argument structures, and baseline performance on causal verification and argument generation. The findings show that extended, structured evidence is central to the data, with event causality, property-based reasoning, and rule-based reasoning as recurring argument forms. Fine-tuned discriminative models perform best on verification, particularly when argument structures are used, while CodeT5+ is strongest on automatic argument-generation metrics, although LLMs appear relatively more competitive in human evaluation and all approaches remain limited.