One-sentence Summary

EPFL proposes fallacy-informed preference optimization (FIPO), which integrates a classification loss over fine-grained fallacy types into preference optimization for LLMs, reducing fallacy errors by up to 17.5% and significantly outperforming DPO and finetuned baselines on argument generation tasks.

Key Contributions

• Fallacy-informed preference optimization (FIPO) is introduced to augment preference optimization with a fallacy-type classification loss, helping large language models generate logically sound arguments.
• A fallacy-aware evaluation setup is constructed for argument generation on the EX-PLAGRAPHS dataset, with GPT-4 serving as an automatic fallacy identifier validated by human annotation and baseline experiments covering zero-shot prompting and retrieval-augmented generation.
• FIPO reduces fallacy errors by up to 17.5% on argument generation tasks, and human evaluation shows that the generated arguments are significantly higher quality than finetuned baselines and preference optimization methods such as DPO.

Introduction

Argument generation supports persuasion in legal, policy, and online debate settings, where logical coherence and evidence quality directly affect trust and decision making. Large language models can draft arguments, but they remain vulnerable to reasoning errors, especially false causality and faulty generalization, and prior work mostly studied fallacy detection or controlled argument generation without teaching models to prefer fallacy free reasoning. Existing preference methods can align arguments with user preferences, yet they often treat all fallacies as equivalent and do not reliably reduce logical errors. The authors leverage a fallacy taxonomy and preference optimization, introducing Fallacy Informed Preference Optimization to add a fallacy classification penalty during training, which improves argument quality and lowers fallacy rates.

Dataset

• The dataset combines three sources:

  • ExplaGraphs, which supplies topics, stances, and short arguments, typically 5 to 20 words, across a broad range of topics.
  • LOGIC, a collection of labeled logical fallacies gathered from diverse web sources. The authors use LOGIC mainly as a reference for fallacy definitions, examples, and a realistic fallacy-type distribution, rather than as the main source of argument samples.
  • Debatepedia, from which a subset is used only for out-of-domain evaluation.

• To build fallacy-aware preference data, the authors start from valid arguments in ExplaGraphs and generate dispreferred alternatives:

  • They define 13 fallacy categories and sample fallacy types from the real-world distribution represented in LOGIC.
  • For each valid argument y_w, ChatGPT generates four equivalent fallacious arguments y_l, each using a different sampled fallacy type.
  • The generation prompts include the definition of the target fallacy and LOGIC examples of that fallacy, which helps filter the outputs and verify that the generated argument is indeed an instance of the intended fallacy.

• The final preference dataset has the schema:

  • topic t
  • stance s
  • preferred valid argument y_w
  • dispreferred fallacious argument y_l
  • fallacy label k for y_l
  • The authors denote the collection as M preferred/dispreferred pairs.

• The authors keep the test set simpler than the training preference set:

  • Test examples are not augmented with generated fallacies.
  • Only topics and stances are used at inference time, so the evaluation measures whether the model can produce logical, non-fallacious arguments on its own.

• The study focuses on short argumentative texts because both ExplaGraphs and LOGIC contain short arguments and fallacies. The paper reports the final train/test split sizes in its dataset table.

Method

The authors leverage a four-stage preference learning framework to align language models with logically valid argumentation. The pipeline begins with supervised fine-tuning, then constructs synthetic preference pairs containing valid and fallacious arguments, applies preference optimization, and finally augments the optimization with fine-grained fallacy classification. This turns fallacy avoidance into an explicit alignment objective rather than treating logical correctness as an unstructured generation preference. As shown in the framework overview:

In the first stage, the authors perform supervised fine-tuning on an argumentation dataset containing topics, stances, and valid arguments. A pretrained language model is optimized with maximum likelihood estimation so that it learns to produce fluent and stance-aligned arguments. For an input consisting of a topic $t$t, a stance $s$s, and a preferred valid argument $y_w$yw​, the supervised training loss is:

This stage produces a base argumentative policy, but supervised training alone does not teach the model why certain arguments should be dispreferred. Therefore, the authors build a preference dataset in which valid arguments are paired with artificially generated fallacious arguments. For each original valid argument, they prompt a generative model to produce an equivalent but fallacious argument under a specified fallacy type. Each generated fallacious argument is conditioned on the topic, stance, a definition of the target fallacy, and representative examples of that fallacy. For each valid argument, the authors generate four fallacious variants with different fallacy types, thereby encouraging broad coverage of logical error categories.

The resulting preference dataset is represented as:

where $y_w^{(i)}$yw(i)​ is the preferred valid argument, $y_l^{(i)}$yl(i)​ is the dispreferred fallacious argument, and $k^{(i)}$k(i) denotes the fallacy type assigned to the dispreferred sample. The preferred arguments are also labeled as non-fallacy examples during the classification stage. The distribution of fallacy types is not assigned uniformly. Instead, the authors use the empirical distribution of fallacies observed in real argumentative texts to guide the sampling of synthetic fallacious arguments. This makes the preference data more representative of natural logical errors. The fallacy distribution used to shape preference data is shown in the figure below:

During preference optimization, the authors use several algorithms, including PPO, DPO, KTO, and CPO. PPO relies on an explicit reward model that is trained on the preference pairs to assign higher rewards to valid arguments and lower rewards to fallacious arguments. The other methods derive implicit rewards directly from preference pairs through contrastive optimization. In general, the aligned policy is encouraged to assign higher probability or higher implicit reward to the preferred argument $y_w$yw​ than to the dispreferred fallacious argument $y_l$yl​.

Although standard preference optimization reduces the overall fallacy rate, the authors observe that it does not explicitly teach the model about the type of logical fallacy present in a rejected argument. This limitation motivates their main method, Fallacy-Informed Preference Optimization, or FIPO. FIPO augments the language model with a classification head that predicts whether an argument is fallacious and, if so, which fallacy type it exhibits. This module operates on the hidden representation produced by the language model. After a forward pass through the model $\pi_\theta$πθ​, the hidden state associated with the last token from the last layer is extracted:

where $L$L denotes the number of layers and $T$T denotes the position of the final token. This representation is passed through a linear classifier followed by a softmax operation. If $k$k is the fallacy type, the predicted probability is:

The classifier treats valid arguments as class $0$0, corresponding to the label “Not a Fallacy.” To ensure that the model prioritizes the most common and practically important fallacy types, the authors define class weights based on their frequency in the preference dataset. Let $w_k$wk​ be the frequency of fallacy type $k$k in $\mathcal{D}'$D′:

The weight for the non-fallacy class is set to the minimum fallacy weight:

This weighting scheme penalizes misclassification of frequent fallacy types more strongly, while preventing the large non-fallacy class from dominating the classification objective. The resulting fallacy classification loss is a weighted cross-entropy loss:

The authors integrate this classification loss into a preference optimization objective. They select CPO as the base preference loss because it offers a favorable balance between argument quality and fallacy reduction. Refer to the human evaluation results comparing preference optimization strategies:

An automatic evaluation using GPT-4 further supports this design choice by jointly examining win-rate and fallacy-rate. Refer to the automatic evaluation results:

Combining the CPO preference loss with the fallacy classification loss yields the final FIPO objective:

Here, $\lambda$λ controls the relative strength of the fallacy-aware classification signal. During training, this objective encourages the model not only to prefer valid arguments over fallacious ones, but also to internalize the specific fallacy type associated with each dispreferred sample.

Experiment

The experiments evaluate short logical argument generation on the EXPLAGRAPHS dataset, with an out-of-domain check on Debatepedia, comparing zero-shot prompting, a supervised fine-tuning baseline, and several preference-optimization methods on small language models. Human annotation and automatic judging are used to compare argument quality and logical soundness, while additional human checks validate GPT-4 as a fallacy classifier and reveal some ambiguity in fallacy categories. Across in-domain and out-of-domain settings, preference-aligned models generally improve over supervised baselines, and the fallacy-aware method yields the strongest overall gains, showing that explicit fallacy guidance, careful loss weighting, and appropriate fallacy distribution are important for producing more logically sound arguments.

Across the evaluated baselines, ChatGPT consistently produces arguments with the lowest fallacy rates in both scenarios, while Llama-2 without retrieval shows the highest error rates. Retrieval augmentation lowers fallacy rates relative to Llama-2, and the second scenario yields lower fallacy rates than the first for all models. ChatGPT achieves the lowest fallacy rates among the baselines Llama-2-RAG reduces fallacies compared with vanilla Llama-2 The second scenario lowers fallacy rates for every evaluated model

The preference dataset is split into training and test portions built from original short arguments and generated fallacies. The training data is substantially expanded with synthetic fallacy examples, while the test portion remains based on the original short-argument data. This supports evaluating fallacy-aware models without testing directly on the generated fallacy instances. The training split combines original argument data with a larger set of generated fallacies. The test split uses only the original short-argument data. The dataset is focused on short arguments and fallacies for fallacy-aware model evaluation.

The agreement scores indicate that human annotators reached moderate reliability when judging argument quality. Agreement between annotators and GPT-4 was very high, suggesting that GPT-4 classifications align well with human judgments in the fallacy evaluation task. Inter-annotator agreement was moderate, consistent with the subjective nature of argument evaluation. Annotator and GPT-4 agreement was high, supporting the use of GPT-4 as a scalable evaluator for fallacy detection.

Ablation experiments show that training FIPO with either a uniformly balanced fallacy dataset or unweighted cross-entropy increases the observed fallacy rate compared with the full method. Preserving the natural imbalance of fallacy types and using weighted cross-entropy is associated with the strongest fallacy reduction. Uniformly sampling fallacy types raises fallacy rates relative to the full FIPO setup. Using unweighted cross-entropy leads to higher fallacy misclassification than weighted cross-entropy. The natural fallacy distribution and weighted classification loss are presented as key contributors to lower fallacy rates.

GPT-4 evaluations show that preference optimization generally lowers logical-fallacy rates compared with the SFT baseline, although DPO improves the least and can perform worse than SFT on Mistral. FIPO achieves the lowest fallacy rates for both Llama-2 and Mistral, indicating that the added fallacy classification signal helps models avoid flawed reasoning. Faulty Generalization is the most frequently observed fallacy type, and FIPO produces the smallest occurrence of it among the compared policies. FIPO produces the fewest fallacies across Llama-2 and Mistral. Alignment methods reduce fallacies relative to SFT, with DPO the weakest improvement. Faulty Generalization is the most common fallacy, and FIPO is especially effective at lowering it.

The experiments evaluate fallacy rates of model-generated arguments across retrieval and scenario settings, using a training set expanded with synthetic fallacies and a test set of original short arguments to assess generalization. Human annotation and agreement with GPT-4 show that GPT-4 can serve as a scalable judge, though argument quality judgments remain somewhat subjective. Ablations indicate that preserving the natural fallacy distribution and using weighted classification loss are important, and comparisons of alignment methods show that preference optimization reduces fallacies overall, with FIPO performing best and DPO weakest, particularly on the common faulty generalization fallacy.