AI flags human errors in complex decisions
Researchers at Cornell University have developed a novel artificial intelligence tool designed to expose and correct human inconsistency in complex decision-making processes. Unlike traditional AI systems that make choices for users, this tool acts as a mirror, analyzing human logic to highlight contradictions between stated values and actual preferences. Created by Abe Davis, an assistant professor of computer science, and Chao Zhang, a doctoral student in information science, the system aims to make evaluation processes more efficient, explainable, and fair. The technology was presented at the Association for Computing Machinery CHI conference on Human Factors in Computing Systems, where it earned the Best Paper Award. Davis conceived the tool while struggling to maintain consistent grading standards for hundreds of student projects, despite having clear criteria and multiple teaching assistants. He identified a fundamental tension in human judgment: people are better at comparing two options directly than rating them on a subjective scale, yet this direct comparison can sometimes mask unconscious biases. The core function of the Interactive Explainable Ranking tool involves a two-step process. First, users define the criteria they value, such as cost or reliability. Next, the AI prompts users to choose between pairs of options to capture their true preferences. The system then cross-references these choices with the initial criteria. If a mismatch is detected, the tool flags the inconsistency. For example, if a user claims fuel efficiency is the most important factor but repeatedly selects less efficient cars, the system will highlight this contradiction. This forces the user to either adjust their ranking, justify their choice with new criteria, or acknowledge hidden biases, such as a preference for a specific color over technical specs. In two case studies, the tool demonstrated its efficacy. In one experiment, participants ranking short films shifted from emotional, intuitive judgments to applying specific criteria after using the system. In another, teaching assistants using the tool to rank student projects produced results that aligned with official grades and showed high consistency among evaluators. Zhang emphasized that the goal is not to outsource decisions to machines, but to use AI to help humans clarify their own priorities. The tool is publicly available, though users can disable the AI function for sensitive applications where algorithmic intervention may be inappropriate. Davis currently uses the tool in his own class, with the AI component turned off, to add rigor to high-stakes grading. He believes the extra effort required to resolve inconsistencies is worthwhile when the value of the decision is significant. The research was published in the Proceedings of the 2026 CHI Conference, with the DOI 10.1145/3772318.3790810.
