Questioning the Role of Intermediate Tokens in Language Model Reasoning: New Study Reveals Surprising Insights

Recent advancements in large reasoning models have led many to view Chain of Thought (CoT) methods as a major success. The idea is that training these models on reasoning traces, often referred to as "thoughts," can help them identify and adopt new reasoning patterns. However, a new study challenges this interpretation by closely examining how the semantic content of these intermediate tokens actually affects model performance. In their research, the authors trained transformer models on formally verifiable reasoning traces and solutions, ensuring that both the intermediate steps and the final outputs aligned with those produced by a formal solver, specifically the A* search algorithm. They developed a formal system to interpret the semantics of the problems and the intended algorithm, allowing them to systematically assess not only the accuracy of the final solutions but also the correctness of the intermediate traces. The findings were intriguing. Despite showing significant improvements over models trained only on correct solutions, the models trained on entirely correct intermediate traces still generated invalid reasoning paths even when they arrived at the right answers. This suggests that the accuracy of intermediate steps may not be as crucial to the final solution as previously thought. To further investigate this point, the authors conducted an experiment where they trained models on noisy, corrupted traces that had no direct relationship to the specific problems they were paired with. Surprisingly, these models maintained performance levels similar to those trained on correct data. In certain cases, they even outperformed models trained on accurate traces and demonstrated better generalization on out-of-distribution tasks. These results challenge the common assumption that intermediate tokens, or Chains of Thought, directly induce predictable reasoning behaviors. The study cautions against anthropomorphizing these outputs or interpreting them too deeply as evidence of human-like or algorithmic reasoning, despite their often correct forms. Instead, the researchers suggest that the success of these models might be attributed to other factors, such as the model's ability to recognize and utilize patterns in the input data, rather than a deep understanding of the reasoning process itself. This research underscores the importance of careful evaluation and interpretation of model outputs. It highlights the need for a more nuanced understanding of how large language models generate solutions, emphasizing that while they can produce impressive results, these outcomes may not always be due to the sophisticated reasoning processes they appear to exhibit. Future work in this area should focus on developing methods to better understand and control the internal reasoning mechanisms of these models, moving beyond the surface-level interpretation of intermediate tokens. Overall, the study provides a critical perspective on the effectiveness of Chains of Thought and encourages the AI community to reconsider how they interpret and leverage these intermediate steps in model training and evaluation.