AI Takes on 30 Mathematicians: Shows Strength in Computation but Still Limited in Logical Reasoning

A group of 30 mathematicians set out to challenge AI, but they nearly failed to stump it. This event, however, does not necessarily indicate that AI truly "understands" mathematics. Jasper, one of the participants, highlighted a key constraint in this challenge: "Each problem requires a specific numerical answer." He explained that this shifted the nature of the questions, making them more suited for computational tasks rather than deep mathematical reasoning. Modern mathematical research often centers on proof and verification, which involve non-trivial computations. Problems can have complex logical structures and deep theoretical underpinnings, but ultimately, the challenge required a precise numerical solution. This change in focus transformed the task into one where an AI could excel through its pattern-matching and computational capabilities, even if the underlying reasoning was flawed. Initially, Jasper and his team designed problems that required advanced logical insights and key theorems. They believed these would expose the weaknesses of current AI models. Surprisingly, the o4-mini model managed to solve most of their proposed problems, but with a caveat. According to Jasper, "Even if the reasoning process was incorrect, it still managed to produce the correct numerical answer." This ability highlights the system's strength in pattern recognition and computation but also underscores its limitations in logical reasoning. The conference revealed other important aspects of AI’s mathematical abilities. Participants observed that o4-mini performed well on problems involving recent research findings, effectively searching and applying the latest academic papers. This capability enhances AI's role in information processing and application, potentially complementing human researchers’ speed and efficiency. However, the successful "counter" problems did emphasize a significant limitation of current large language models (LLMs): they struggle with multi-step, intricate logical reasoning and the synthesis of novel concepts. These tasks highlight the gap between AI systems and human creativity and deep logical integration. In summary, AI has indeed made remarkable progress over the past two years, but current LLMs still rely heavily on pattern matching and computational power. They lack the ability to generate entirely new mathematical results, though they excel at gathering relevant data and proposing initial solutions. Human oversight remains crucial, especially in verification and synthesis, as these areas cannot be overlooked. Looking ahead, Jasper predicts that within the next two years, AI will primarily serve as a "research assistant," aiding mathematicians in discovering new theories and solving open problems, much like the collaboration between DeepMind and leading mathematicians. Eventually, AI may work as a "collaborator" before independently pushing the boundaries of mathematical research. References: 1. https://www.scientificamerican.com/article/inside-the-secret-meeting-where-mathematicians-struggled-to-outsmart-ai/ 2. https://x.com/zjasper666/status/1931481071952293930 整理：槐树