One-sentence Summary

LMU Munich, MCML, and The University of Tokyo introduce IMPERFECTIVENLI, a diagnostic dataset probing the Imperfective Paradox, which reveals that open-weight LLMs exhibit a pervasive Teleological Bias by systematically hallucinating event completion for goal-oriented accomplishments, indicating they operate as predictive narrative engines rather than faithful logical reasoners.

Key Contributions

• IMPERFECTIVENLI is a diagnostic natural language inference dataset that tests whether open-weight LLMs respect the Imperfective Paradox across diverse semantic classes, including atelic activities that entail completion and telic accomplishments that do not.

• Evaluations reveal a pervasive Teleological Bias, where models systematically hallucinate event completion for goal-oriented predicates and can override explicit textual cancellation, conflating ongoing processes with achieved results.

• Prompting interventions only partially reduce these hallucinated completions and can cause a calibration crisis that rejects valid entailments for atelic verbs, while representational analysis shows embeddings may distinguish progressive from simple past forms but inference is dominated by goal-attainment priors.

Introduction

Large language models are increasingly expected to support natural language inference over temporal events, where reasoning must separate an action in progress from its completed outcome. This matters because goal-oriented events such as building or writing have a culmination point, while activities such as running do not, yielding different logical entailments.

Prior work often modeled aspect by classifying verbs or studying human judgments, but it did not systematically test whether generative LLMs can resist invalid entailments in zero-shot reasoning. This leaves a gap around whether apparent aspectual understanding reflects genuine semantic reasoning or probabilistic shortcuts.

The authors leverage this distinction to introduce ImperfectiveNLI, a diagnostic dataset of minimal pairs contrasting interrupted and ambiguous contexts for telic and atelic verbs. They use it to evaluate open-weight models and identify a teleological bias toward inferred completion, a calibration issue in prompting, and a dissociation between aspect representations and inference.

Dataset

• The authors use IMPERFECTIVENLI, a diagnostic English NLI dataset designed to test whether large language models can reason about the imperfective paradox and teleological aspectual semantics.

• Dataset sources and construction:

  • The dataset is built with template-based generation to keep syntax uniform while isolating semantic reasoning.
  • The authors curate a balanced verb lexicon based on Vendler’s aspectual classification:
    • 100 telic verbs, or accomplishments, that involve a natural endpoint or result state, such as build, write, fix, paint, and bake.
    • 100 atelic verbs, or activities, that describe homogeneous processes without a necessary endpoint, such as run, swim, wander, play, and vibrate.
  • Context-hypothesis pairs are generated with Gemini-assisted rewriting and then strictly verified by humans.
  • Total size: 400 examples, created as controlled minimal pairs across four logical conditions.

• Schema and task format:

  • Each example contains a premise and a hypothesis in an NLI-style pair.
  • Standard NLI labels are used:
    • Entailment is mapped to “True”.
    • Contradiction is mapped to “False”.
    • Neutral is mapped to “Unknown”.

• Four dataset subsets:

  • Group A: Interrupted Accomplishment
    • Uses telic verbs with explicit cancellation clauses, such as a failure or interruption before completion.
    • Logical label: Contradiction.
    • Tests whether models recognize that the action was interrupted and the result did not occur.
  • Group B: Interrupted Activity
    • Uses atelic verbs with explicit interruption clauses.
    • Logical label: Entailment.
    • Exploits the sub-interval property of activities: if someone was stopped while running, it still follows that they ran.
  • Group C: Ambiguous Accomplishment
    • Uses telic verbs in progressive form without an interruption clause.
    • Created as an ablation of Group A.
    • Logical label: Neutral.
    • Tests whether models hallucinate completion when outcome evidence is absent.
  • Group D: Ambiguous Activity
    • Uses atelic verbs in progressive form without an interruption clause.
    • Created as an ablation of Group B.
    • Logical label: Entailment.
    • Tests basic aspectual competence for atelic verbs.

• Additional annotation and filtering details:

  • The authors annotate the 100 telic verbs into four finer semantic classes:
    • Change of State: 44 verbs, such as melt and break.
    • Creation: 39 verbs, such as build and knit.
    • Consumption: 9 verbs, such as eat and burn.
    • Motion to Goal: 8 verbs, such as arrive and enter.
  • This sub-classification applies only to telic verbs because they encode a distinct resultant state.
  • The purpose is to test whether models are more likely to infer completion for goal-directed or artifact-creating events than for ordinary state changes.

• Quality assurance:

  • Three native English evaluators recruited through Prolific review the sentences for Grammar, Fluency, and Adequacy.
  • Evaluators provide corrections for unnatural or unclear instances.
  • Conflicting judgments and revisions are resolved through manual verification before the final dataset is retained.

• How the authors use the data:

  • The authors use IMPERFECTIVENLI as a controlled diagnostic evaluation set rather than a large-scale training resource.
  • It is organized as a controlled minimal-pair experiment crossing verb telicity, telic versus atelic, with contextual information, interrupted versus ambiguous.
  • The dataset is used to probe model reasoning about whether ongoing actions entail, contradict, or leave unknown whether their results occurred.

Method

The authors construct IMPERFECTIVENLI as a controlled diagnostic evaluation pipeline for testing language models on imperfective paradox reasoning. Rather than relying on open-ended text samples, the pipeline uses template-based generation to create syntactically uniform context-hypothesis pairs while isolating the aspectual semantics of interest. The central design principle is minimal variation: only the relevant aspectual variables are changed across examples, allowing model behavior to be attributed to specific semantic factors.

The first stage of the pipeline builds a balanced verb lexicon based on Vendler’s aspectual classification. The authors curate 100 telic verbs, corresponding to accomplishments or other predicates that imply a natural endpoint or result state, such as build, write, fix, paint, and bake. They also curate 100 atelic verbs, corresponding to activities that describe homogeneous processes without a required endpoint, such as run, swim, wander, play, and vibrate. This separation is essential because the imperfective paradox concerns whether an ongoing process entails completion. For telic predicates, the process and the completion are conceptually distinct, whereas for atelic predicates, the process itself constitutes the event.

For each verb, the authors generate context-hypothesis pairs using assisted rewriting, followed by strict human verification. This hybrid generation and verification workflow helps ensure that the sentences remain natural while preserving the intended logical structure. The resulting dataset is organized as a controlled minimal-pair experiment with a total of 400 examples. The examples are mapped onto four logical conditions formed by crossing verb telicity with contextual information. The two main dimensions are Verb Telicity, namely Telic versus Atelic, and Contextual Information, namely Interrupted versus Ambiguous. This design produces a $2 \times 2$2×2 experimental structure that separates the effects of lexical aspect from the effects of contextual completion cues.

The method also introduces a finer-grained semantic analysis of the telic group. The 100 telic verbs are further annotated into four classes: Change of State, Creation, Consumption, and Motion to Goal. This stratification allows the authors to examine whether models exhibit different levels of teleological bias depending on how salient the goal or endpoint is. Change of State verbs denote transformations in the condition of an existing entity, such as melt or break. Creation verbs involve bringing an entity into existence, such as build or knit. Consumption verbs entail destruction or complete ingestion, such as eat or burn. Motion to Goal verbs describe movement toward a specific endpoint, such as arrive or enter. This distinction is important because completion hallucination may be stronger for events where the endpoint defines the meaning of the verb, such as creation or consumption, than for more abstract state changes.

The evaluation labels are aligned with standard natural language inference categories but are represented through natural language truth values. Entailment is mapped to True, contradiction is mapped to False, and neutral is mapped to Unknown. This mapping supports experiments that ask models to judge whether a hypothesis is entailed by a premise in cases where progressive aspect interacts with event completion. For example, in ambiguous atelic activity conditions, the entailment is expected to hold because the ongoing process validates the event described by the activity predicate. This condition serves as a baseline for basic aspectual competence.

Quality assurance is incorporated as a final filtering stage. Three native English speakers evaluate the generated sentences across grammar, fluency, and adequacy. Evaluators provide both quantitative judgments and corrections for unnatural or unclear sentences. Conflicting revisions are resolved through manual verification, producing a final dataset that balances controlled experimental structure with linguistic naturalness.

Experiment

The evaluation uses seven 7B to 9B instruction-tuned open-weight models on imperfective NLI with zero-shot and three targeted prompts, measuring accuracy, completion hallucination, and aspectual awareness gaps. The experiments show that models often infer completion from progressive forms, indicating robust teleological bias, while explicit rules or reasoning prompts only partially mitigate it and can impair atelic entailments or cause over-skepticism. Scaling analysis suggests aspectual discrimination emerges strongly with model size, and verb-semantic and representation analyses indicate that creation and other telic events induce stronger completion priors, with failures arising mainly at reasoning rather than encoding.

The examples show that progressive sentences with telic verbs do not logically entail completed events, while atelic progressive sentences do. The reported model behavior suggests that large language models often infer completion from ongoing goal-directed actions, especially for creation events. This pattern appears to reflect a teleological bias during reasoning rather than a failure to encode the distinction between progressive and perfective forms. Telic actions described in the progressive are presented as lacking entailment of their completion, unlike atelic actions. Models show stronger completion bias for creation verbs than for motion to goal verbs, even when the representation distinguishes ongoing from completed forms.

The diagnostic examples compare telic and atelic events under interrupted and ongoing conditions to expose reasoning about completion. They show that interrupted telic events should be labeled Contradiction, interrupted atelic events Entailment, and ambiguous telic events Neutral, making the ambiguous telic case a sensitive probe of teleological bias. The accompanying results indicate that models are most accurate with motion-to-goal verbs and most prone to infer completion for creation verbs, even when their internal representations distinguish progressive from perfective forms. Ongoing creation events are marked Neutral because the imperfective premise does not guarantee that the goal was reached. Creation verbs produce stronger completion hallucinations than motion-to-goal and change-of-state verbs. The bias appears at inference rather than encoding, since models still overpredict completion despite separating process and result in representations.

Across zero-shot prompting, the seven open-weight models show a consistent separation between activity-like and accomplishment-like judgments: they are generally strong on interrupted activities and ambiguous activity cases, but weak on interrupted accomplishments and ambiguous accomplishment cases. The very small Delta_AA and high TBR_C for most models indicate that these systems largely use a default pattern in which an ongoing event is inferred to have reached its result, rather than genuinely tracking atelic versus telic aspect. Qwen2.5-7B-Instruct deviates from this pattern with better accomplishment accuracy and a larger aspectual gap, though it still trails its performance on activity items. Models achieve high accuracy on activity items while failing sharply on ambiguous accomplishment items. Near-zero Delta_AA values for most models show little ability to distinguish atelic from telic progressive interpretations. Interrupted accomplishment scores are low overall, suggesting that explicit cancellation is often overridden for telic events. Qwen2.5-7B-Instruct has the clearest aspectual separation and lower teleological bias among the zero-shot models.

Experiments use diagnostic telic and atelic progressive cases, including interrupted, ongoing, and ambiguous events, to test whether language models track aspectual distinctions and infer completion only when warranted. Across zero-shot open-weight models, models generally handle atelic activity cases well but overpredict completion for telic accomplishments, especially creation verbs, suggesting a teleological inference bias rather than a basic inability to represent ongoing versus completed forms. Qwen2.5-7B-Instruct shows a clearer aspectual separation and weaker completion bias, but most models still default toward assuming ongoing goal-directed actions reach their endpoints.