One-sentence Summary

Researchers from MOEKLINNS at Xi’an Jiaotong University, WeChat AI, and Tencent Inc. propose LoCo, a local preference learning method that detects candidate overthinking states, regenerates shorter verified continuations, and applies state-conditioned DPO to teach models when to conclude, cutting completion lengths by $41.7\text{--}45.4\%$41.7–45.4% on mathematics and $23.4\text{--}36.3\%$23.4–36.3% on code while maintaining or improving accuracy.

Key Contributions

• LoCo is a local preference learning method that constructs training data entirely from self-generated mathematical traces by detecting candidate overthinking states, freezing the corresponding reasoning prefixes, and regenerating shorter verified continuations from the same states.
• LoCo formulates reasoning compression as learning state-conditioned continue-or-conclude preferences with DPO, teaching the model when to conclude or continue at specific reasoning states rather than globally favoring shorter trajectories.
• On five mathematical and three code generation benchmarks, LoCo reduces completion length by 41.7% to 45.4% on mathematics and 23.4% to 36.3% on code while maintaining or improving accuracy, and it achieves the best AUC across all code benchmarks. LoCo also transfers from math-only training to code generation, indicating cross-domain generality.

Introduction

Large reasoning models achieve strong results on complex mathematical and coding tasks through extended chain-of-thought reasoning, but this creates high inference latency and memory cost, and longer reasoning does not always improve accuracy because models often overthink after already reaching a sufficient state. Prior compression methods either compare complete trajectories, which gives a global preference for shorter solutions but cannot localize unnecessary reasoning, or apply fine-grained edits that remove or shorten content without directly modeling whether further reasoning is needed from the current state. The authors introduce LoCo, a local preference learning method that identifies overthinking states in self-generated traces, constructs state-conditioned preferences by contrasting verified continuations from the same reasoning prefix, and learns continue-or-conclude behavior with DPO. Evaluated on mathematical reasoning and code generation benchmarks, LoCo improves the efficiency-performance trade-off and transfers across domains by reshaping reasoning preference rather than adding task-specific knowledge.

Method

The authors leverage a method called LoCo to convert a reasoning model's own overthinking traces into state-anchored preference supervision. This process operates through three sequential stages: local overthink detection, verified continuation generation, and local preference learning.

As shown in the figure below:

The first stage, step-level overthink detection, identifies reasoning states where the model begins to overthink by analyzing its predictive confidence. The authors feed the complete chain of thought back to the model and score it under its own next-token distribution. For each step, they calculate the step confidence by averaging the maximum next-token probabilities across all tokens in that step. They also measure confidence volatility over a sliding window of the most recent steps. Overthinking states are identified as those exhibiting simultaneously low confidence and high volatility, specifically where the step confidence falls below the 25th percentile and the volatility exceeds the 75th percentile of the model's self-generated traces. To ensure reliability, a stronger judge model re-examines each flagged state. The judge evaluates the current reasoning state and the subsequent step, classifying it as redundant overthinking, normal progress, or a final answer. Only states confirmed as redundant overthinking are retained, ensuring that useful progress or final answer formatting is not discarded.

In the second stage, verified continuation generation, the authors freeze the reasoning state for each retained detection point and sample multiple independent continuations from the model. A continuation is deemed valid if it completes the reasoning template and produces an answer symbolically equivalent to the ground truth. Among all valid continuations, the shortest one is selected as the preferred continuation. The original continuation from the same state is designated as the rejected continuation. The detection point is kept only if the regenerated continuation is strictly shorter than the original.

The final stage, local preference learning, constructs preference triples from the surviving detection points. The authors apply further filtering to ensure high-quality training data. A pair is retained only if the token savings reaches a minimum threshold, and both the prompt and continuations satisfy specific token length constraints. Additionally, to prevent questions with numerous detection points from dominating the dataset, at most a fixed number of pairs with the largest token savings are kept per question.

With the filtered dataset, the authors optimize a Direct Preference Optimization objective. Since both the chosen and rejected continuations share the identical frozen prefix, the preference comparison is strictly conditioned on the same reasoning state. The loss function is defined as:

where $\pi_{\text{ref}}$πref​ is the frozen reference policy, and the log-likelihoods are computed as the sum of token log-probabilities over the respective continuations. This formulation teaches the model a local continue-or-conclude preference, guiding it to terminate with a shorter, valid continuation rather than proceeding with redundant reasoning once it reaches a specific state.

Experiment

The paper evaluates LoCo, which trains a LoRA adapter with DPO on self-generated math reasoning traces and local compression cutpoints, across mathematical reasoning and code generation benchmarks. The main evaluations show that LoCo transfers from math to coding, matching or improving accuracy and AUC while substantially reducing generated tokens, with the strongest gains on harder tasks where retaining difficult reasoning matters most. Ablations confirm that local cutpoints are critical for compression, self-generated continuations outperform external ones due to better distributional alignment, and detector-selected compression points preserve accuracy better than random or uniform alternatives.

LoCo achieves the strongest token reduction across the reported mathematical reasoning benchmarks while preserving or improving accuracy relative to the base model. Its advantage grows on harder tasks, where it improves both accuracy and AUC despite cutting completion length by over 40 percent. Compared with DAST and TokenSqueeze, LoCo provides the best overall compression-accuracy trade-off. LoCo reduces generated tokens by more than 40 percent across the reported benchmarks, the largest reduction among all methods. On harder benchmarks, LoCo improves accuracy and AUC over the base model while compressing, whereas DAST can increase completion length and TokenSqueeze tends to lose accuracy as difficulty rises.

LoCo transfers its learned compression preference from math-only training to code generation. It matches base accuracy on MBPP while producing substantially shorter completions and improving AUC, and on harder coding benchmarks it outperforms all baselines in accuracy and AUC while using fewer tokens. This pattern contrasts with DAST, which increases token use and loses accuracy, and TokenSqueeze, which compresses but with more limited accuracy benefits. LoCo matches base accuracy on MBPP while producing substantially shorter completions and improving AUC. On the more difficult LiveCodeBench and CodeContests benchmarks, LoCo improves accuracy and AUC over all baselines while reducing token usage.

The ablations show that both local preference construction and detector-selected cutpoints contribute to LoCo's stronger compression while preserving accuracy. Replacing local state-conditioned preferences with global trajectory-level preferences results in only modest token reductions. Random cutpoints can shorten reasoning but lose accuracy on harder benchmarks, and self-generated continuations are better aligned than external continuations. Local preference construction drives large token reductions; global trajectory-level preferences compress far less despite similar accuracy. Detector-selected cutpoints avoid the accuracy losses seen with random cutpoints, and self-generated continuations outperform external continuations.

The experiments evaluate LoCo on mathematical reasoning and code generation benchmarks, along with ablations of its compression components. On math tasks, LoCo achieves the largest token reduction among compared methods while preserving or improving accuracy and AUC, with gains becoming clearer on harder benchmarks. It also transfers to code generation, matching base accuracy on MBPP with shorter outputs and outperforming baselines on harder coding tasks while using fewer tokens. Ablations show that local preference construction and detector-selected cutpoints drive compression without accuracy loss, and self-generated continuations align better than external ones.