Large language models like those powering OpenAI’s ChatGPT frequently produce confident but incorrect statements—known as hallucinations—despite being trained on vast amounts of data. A recent preprint by researchers from OpenAI and the Georgia Institute of Technology reveals that these errors aren’t just due to flawed training data. Instead, the root cause lies in how models are trained and evaluated. Even with perfect data, some questions are inherently unanswerable, making perfect accuracy impossible. Yet models still generate answers rather than admitting uncertainty. The core issue stems from the way models are fine-tuned after pretraining. During this phase, human feedback and benchmarking shape model behavior. Most popular benchmarks score answers as either correct (1) or incorrect (0), with no penalty for guessing wrong. This creates a perverse incentive: models learn to bluff rather than say “I don’t know,” because confident wrong answers often score better than honest uncertainty. The researchers argue that benchmarks must be redesigned to penalize incorrect guesses more heavily than nonanswers. This shift would encourage models to recognize their limits and prioritize truthfulness over engagement. One proposed solution is to reward models not just for correct answers, but for knowing when they don’t know—teaching them humility through a kind of “school of hard knocks.” Experts have mixed reactions. Some, like Princeton’s Carlos Jimenez, support the idea, noting it’s a relatively simple change that could improve model reliability. Others, like University of Illinois’ Hao Peng, remain skeptical, warning that models are already adept at gaming any system they’re optimized for. Incentivizing “I don’t know” could lead to unintended behaviors, such as overuse of the phrase to game the system. The real challenge, however, is not just technical but economic. Companies like OpenAI are under pressure to grow users and revenue. If models frequently say “I don’t know,” users may turn to more confident competitors, even if those models are less accurate. As Delft University’s Servaas Storm points out, with rising costs and diminishing returns, no company wants to be the first to break the status quo. Arizona State’s Subbarao Kambhampati puts it bluntly: “If LLMs keep pleading the Fifth, they can’t be wrong. But they’ll also be useless.” The tension between truthfulness and usability remains unresolved. While the research offers a clear path forward, its success depends on whether AI companies are willing to sacrifice short-term performance for long-term integrity.