New AI Detection Tool Achieves High Accuracy with Low False Accusation Rate, Aiding Academia and Policy Making

Researchers at the University of Michigan have developed a new tool called "Liketropy" that aims to accurately detect text written by Large Language Models (LLMs) while minimizing false accusations of human-written content. This dual objective addresses a significant challenge in the field of AI detection, where tools often err either by missing AI-generated content or by wrongly flagging human work. Liketropy combines two statistical concepts—likelihood and entropy—to create what the researchers describe as "zero-shot statistical tests." These tests can distinguish between human and AI-generated text without needing prior training on samples of each type. By analyzing the predictability and surprise factor of the words in a given text, Liketropy determines whether the content is more likely to have been produced by a human or an AI. In extensive testing using large-scale datasets, including those where the underlying AI models were not publicly known or where AI-generated text was designed to evade detection, Liketropy demonstrated impressive performance. When tailored specifically for known LLMs, the tool achieved an average accuracy of over 96% and maintained a false positive rate of just 1%. Tara Radvand, a doctoral student at the University of Michigan’s Ross School of Business and a co-author of the study, emphasized the importance of avoiding overconfidence in AI detection tools. She noted that false accusations, especially in educational and policy contexts, can have serious repercussions. The team's approach is cautious and transparent, aiming to strike a balance between accuracy and fairness. One of the unexpected outcomes of the research was discovering that very little knowledge about the language model was required to effectively detect AI-generated content. This challenges the common belief that AI detection must rely heavily on training, access, or cooperation with the model developers. The findings suggest that Liketropy could be widely applicable and adaptable to different settings. The researchers are particularly concerned about the implications for international students and non-native English speakers, who might be unfairly flagged by existing AI detectors due to differences in tone and sentence structure. Liketropy offers a solution by allowing these students to self-check their work in a low-stakes environment, ensuring that their writing is not mistakenly identified as AI-generated. Looking forward, the team plans to expand Liketropy into a versatile tool that can be tailored to various fields, such as law, science, and college admissions, where the balance between caution and effectiveness may vary. They aim to collaborate with University of Michigan (U-M) leaders to integrate their tool with existing AI assistants like U-M GPT and Maizey to verify the origin of text submitted through these channels. The application of AI detectors like Liketropy in combating misinformation on social media is another critical area. By promptly identifying false content andcomments generated by LLMs, platforms can limit the spread of harmful narratives and safeguard public discourse. Early detection is essential for maintaining the integrity of online information. The research received a Best Presentation Award at the Michigan Student Symposium for Interdisciplinary Statistical Sciences and was featured by Paris Women in Machine Learning and Data Science, a community promoting women in these fields. The team’s work is available on the arXiv preprint server. Industry insiders highlight the tool’s potential to address the growing concern over AI-generated content in academic and professional settings. They believe that Liketropy’s ability to balance accuracy and fairness makes it a promising addition to the toolkit for managing the ethical and practical challenges posed by AI. The University of Michigan’s commitment to developing innovative solutions in AI ethics and detection further solidifies its position as a leader in this rapidly evolving field.