Decoupling AI Confidence and Correctness Enhances Answer Reliability

Researchers at the University of California, Riverside, led by doctoral student Het Patel, have demonstrated that confidence and correctness in large language models originate from separate internal features, challenging the prevalent assumption that high confidence guarantees accuracy. Their findings suggest developers can enhance AI reliability by isolating and manipulating these mechanisms post-training, rather than relying on costly model retraining. Using sparse autoencoders to inspect Meta's Llama-3.1-8B and Google's Gemma-2-9B, the team analyzed model behavior across multiple-choice benchmarks. They identified three distinct categories of internal features: those driving uncertainty, those linked to incorrect answers, and confounded features associated with both. Suppressing features tied purely to uncertainty significantly degraded accuracy, while disabling incorrectness-related features had minimal impact. Crucially, targeting confounded features allowed the team to improve accuracy by up to 1.1 percent and reduce uncertainty by 75 percent without altering the model's weights. This separability enables a novel approach to error mitigation. By monitoring specific confounded features within a middle layer of the Llama model, researchers could predict impending errors with high precision. Implementing a system where the model abstains from answering flagged queries increased accuracy from 62 percent to 81 percent while retaining responses for approximately 53 percent of inputs. This intervention significantly outperformed standard open-ended "I don't know" options, which only raised accuracy to roughly 64 percent. The research indicates that these internal signals are generalizable across benchmarks and offer a practical pathway for developers to adjust model behavior dynamically. The methodology could extend beyond confidence and correctness, allowing teams to target other specific behaviors by modifying related internal activations. Treating internal features like adjustable knobs provides a cost-effective strategy for building AI systems that can better recognize their limitations and self-regulate responses.