One-sentence Summary

Researchers from Technion and the University of Utah propose Faithfulness by Unlearning Reasoning steps (FUR), which erases information in reasoning steps from model parameters to measure parametric faithfulness of chain-of-thought, and experiments on four LMs and five multi-hop MCQA datasets show that unlearning key steps frequently alters predictions while post-unlearning CoTs support different answers.

Key Contributions

• The paper proposes a parametric faithfulness framework (PFF) and Faithfulness by Unlearning Reasoning steps (FUR), which erases reasoning-step information from model parameters through unlearning and measures faithfulness by the resulting effect on model predictions. FUR uses generated chains of thought to guide unlearning while producing predictions without chains of thought, helping separate parametric influence from contextual prompting.

• The work defines FF-HARD and FF-SOFT to quantify chain-of-thought faithfulness: FF-HARD tests whether a generated chain of thought is faithful, while FF-SOFT identifies which reasoning steps are most relevant to the model’s prediction.

• Experiments with four language models and five multi-hop multiple-choice QA datasets show that unlearning key steps often changes the model’s prediction and leads post-unlearning chains of thought to support different answers, while steps identified as highly relevant are not always judged plausible by humans.

Introduction

Language models increasingly use chain-of-thought reasoning to solve complex tasks and make their outputs easier to interpret, but it is unclear whether those verbalized steps actually reveal the model’s internal reasoning. Prior studies usually test faithfulness by perturbing tokens in the chain and checking whether predictions change, yet this mainly measures contextual consistency because the original knowledge remains in the model’s parameters and can be reconstructed during prediction. The authors introduce a Parametric Faithfulness Framework that intervenes directly on model parameters and then tests whether removing reasoning-related knowledge changes the final answer. They instantiate this framework with a machine unlearning method called FUR and propose FF-HARD and FF-SOFT metrics to assess whether an entire chain of thought, or individual reasoning steps, are faithful to the model’s underlying prediction.

Dataset

• The authors construct intervention datasets from chain-of-thought reasoning steps in the dataset used for unlearning, rather than from a newly introduced standalone corpus.

• The source pool is the original dataset containing instances with CoT steps, and data are organized around individual reasoning steps.

• A reasoning step is filtered as a sentence with at least two content words before it is used.

• For each target reasoning step, the forget set is built as input-output pairs. For every content word in the step, the input is the prefix up to that word, and the output is the token sequence of that content word.

• The retain set for a target step is constructed from four randomly selected CoT steps, each with at least two content words, taken from four other instances in the same dataset.

• Retain data use the content words from those randomly selected steps as target tokens, while the original model outputs on these contexts are preserved through KL regularization.

• The resulting data are used step by step: each reasoning step is unlearned individually for five iterations, with the forget set containing prefixes and content-word outputs and the retain set containing content-word targets from unrelated steps.

Method

The authors formulate parametric faithfulness as a two phase experimental pipeline. The first phase targets the model parameters, while the second phase evaluates whether generated reasoning is linked to the computation stored in those parameters. As shown in the framework diagram:

The intervention phase begins by prompting the original model $\mathcal{M}$M to produce a reasoning chain. This chain is divided into reasoning steps at a chosen granularity. Each step is then treated as the target of a parameter level intervention. The goal is not to alter the visible prompt or the context, but to modify the parameters where the information expressed by that reasoning step is assumed to be stored. After the intervention, the model becomes $\mathcal{M}^{*}$M∗, or, when a single step is unlearned, $\mathcal{M}^{(i)*}$M(i)∗. Because the success of the whole measurement depends on whether the parameter update truly reflects the intended information erasure, the framework also includes controls that verify the intervention. The evaluation phase then compares the original and modified models. The authors consider two protocols. In the first, both models are asked to answer directly, and faithfulness is inferred from changes in their predictions. In the second, both models are asked to generate reasoning before answering, and faithfulness is inferred from changes in the reasoning content. The central intuition is that if verbalized reasoning is faithful to internal computation, removing that computation from the parameters should produce a measurable behavioral difference.

They instantiate this framework through unlearning reasoning steps. For each generated chain, they first identify reasoning steps as sentences containing at least two content words. For a given step, they construct a forget set $\mathcal{D}_{\mathrm{FG}}$DFG​ by taking each content word, using the prefix up to that word as input, and using the content word tokens as output. The model is then trained to discourage these target completions. To preserve general fluency and avoid broad degradation, they also construct a retain set $\mathcal{D}_{\mathrm{RT}}$DRT​ from content words taken from other chain of thought steps sampled from other instances. The retain loss encourages the unlearned model to preserve the original model’s behavior on unrelated in domain contexts. The authors use the KL regularized form of preference optimization unlearning, which updates parameters to reduce likelihood on forgotten content words while constraining the output distribution on retained content words through a KL divergence term.

The training procedure is deliberately targeted. Instead of fine tuning all parameters, the authors modify only the second feed forward matrix in the transformer MLP blocks, treating these matrices as likely storage locations for factual or procedural knowledge. For each reasoning step, the intervention is run for five unlearning iterations, with each iteration corresponding to one pass over the forget set. In this setup, one reasoning step is unlearned independently, producing a modified model $\mathcal{M}^{(i)*}$M(i)∗ that should no longer rely on the erased step for prediction if the original reasoning was faithful to the internal computation.

The validity of the intervention is checked along three dimensions: efficacy, specificity, and preservation of general capability. Efficacy measures how strongly the probability of the target reasoning step has decreased after unlearning. For a reasoning step $r_i$ri​ with $T$T tokens and prefix $\mathrm{pf}_i$pfi​, the length normalized probability under a model $\mathcal{M}$M is

The efficacy of unlearning step $i$i is

This score captures how much the original chain of thought step has become a less likely reasoning path after the parameter intervention. It is intentionally normalized, and the authors note that perfect efficacy, corresponding to probability zero, is not necessarily desirable because it can harm fluency.

Specificity checks whether unlearning has damaged behavior on unrelated instances drawn from the same task distribution. Using a held out set $\mathcal{D}_s$Ds​, the authors compare predictions before and after unlearning and compute the proportion of unchanged labels:

This score helps distinguish meaningful knowledge erasure from broad model degradation. General capabilities are assessed separately, using zero shot MMLU style evaluation after unlearning a subset of reasoning steps. The authors do not require MMLU performance to remain identical, but they use it to confirm that the targeted parameter updates have not produced severe global damage.

After verifying that the intervention is precise, faithfulness is measured through behavioral differences between the original model and the modified model. The hard measure answers whether any unlearned step flips the most likely final answer:

This produces a binary faithfulness signal for a full reasoning chain. If removing step $r_i$ri​ from the model’s parameters changes the final prediction, that step can be treated as causally linked to the original answer. The soft measure assigns a continuous value to each step by comparing the probability assigned to the original answer before and after unlearning that step:

This score identifies which reasoning steps carry larger shares of the predictive evidence. The hard score is interpreted as a lower bound on true faithfulness, because some faithful steps may still not change the final answer if the model relies on multiple valid reasoning paths or if the intervention does not fully locate the causal storage site.

To make the intervention feasible for longer reasoning chains and larger models, the authors explore a parameter efficient variant of the unlearning update. Instead of updating the full second feed forward matrices, they apply LoRA adaptation to the same target matrices. This reduces the number of active trainable parameters while keeping the intervention localized. The comparison between full updates and LoRA ranks 8, 32 and 128 is shown in the following figure:

The comparison reports FF-HARD scores for full fine tuning and the LoRA variants. This part of the design shows how the core parameter intervention can be adapted to lower memory cost by restricting unlearning to low rank subspaces of the same architectural components that would otherwise be updated.

Experiment

The experiments use zero-shot multi-choice question answering across several datasets and instruction-tuned models to measure whether unlearning selected chain-of-thought steps reduces target information while preserving held-out accuracy and general model capabilities. These controls show that the unlearning intervention is precise, and further analyses show it frequently changes both predictions and verbalized reasoning, with stronger unlearning generally indicating greater faithfulness compared with contextual perturbation baselines. Step-level analysis and human evaluation reveal that steps most influential for predictions are not always perceived as most plausible, while preliminary scaling experiments suggest low-rank adaptation may make the approach more practical.

The results show that unlearning lowers the probability assigned to targeted chain of thought steps while leaving held out agreement and post unlearning MMLU accuracy broadly stable. Efficacy varies by dataset and model, with stronger reductions on some tasks and weaker reductions on others, but the models generally retain their specificity and general capability. Unlearning is effective at reducing confidence in targeted reasoning steps without substantially harming held out agreement or general MMLU accuracy. Mistral-2 shows relatively strong reduction across several datasets, while Phi-3 maintains the highest baseline and post unlearning MMLU accuracy. Efficacy is least consistent on StrategyQA and TruthfulQA, where some models show much smaller reductions than on ARC Challenge or OpenbookQA.

Across the evaluated models and datasets, removing information from a reasoning step changes the model answer more often than introducing a mistake into that step, suggesting that parametric unlearning can identify faithful reasoning chains more broadly than contextual error injection. The results still show meaningful prediction changes under both interventions, with some dataset-specific cases where mistake addition performs comparably or higher, likely due to knowledge overlap across instances. Unlearning a reasoning step leads to answer changes more frequently than adding a mistake to that step. Both interventions produce substantial prediction changes on instances where chain-of-thought and direct predictions agree. Stronger unlearning is associated with more frequent prediction changes, supporting the view that generated reasoning chains are often faithful.

After unlearning, chain-of-thought reasoning frequently supports an answer different from the base model's original prediction, indicating that the intervention changes verbalized reasoning rather than only surface output. The trend appears across all evaluated models and datasets, with most majority rates well above chance, though some benchmark and model pairs show more modest shifts. The pattern suggests unlearning can propagate into downstream rationale for most tested settings. Post-unlearning reasoning commonly diverges from the base model's original answer Reasoning changes are observed across both model sizes and all benchmark categories Some datasets and models show weaker reasoning shifts than others

Experiments evaluate parametric unlearning of targeted chain-of-thought steps across models and datasets, showing that it reduces confidence in specific reasoning while preserving held-out agreement and general MMLU accuracy, with model-dependent and dataset-dependent efficacy. They further compare unlearning with contextual error injection and find that removing reasoning-step information changes final predictions more reliably, supporting the faithfulness of generated reasoning chains and the specificity of unlearning. Finally, post-unlearning explanations often shift away from the original answer, indicating that the effect propagates through verbalized reasoning rather than only altering surface outputs.