As new versions of artificial intelligence (AI) language models are frequently released, developers often claim enhanced performance. However, proving that a newer model is genuinely superior to its predecessor is a complex and costly endeavor. The typical method involves testing these models with a vast array of benchmark questions, which must be manually reviewed by humans, significantly increasing the time and expense. Due to practical limitations, only a subset of these questions is used, which can lead to inaccurate assessments if the chosen questions are too easy or too difficult. Stanford researchers, led by Assistant Professor Sanmi Koyejo and doctoral candidate Sang Truong, have introduced a more cost-effective and fair method for evaluating AI language models, detailed in a paper published at the International Conference on Machine Learning (ICML 2025) and available on the arXiv preprint server. The team's key insight is that question difficulty must be factored into the evaluation process to avoid overestimating a model's capabilities or underestimating its weaknesses. Koyejo explains that the approach they've adopted, Item Response Theory (IRT), has been used in education for decades to score test-takers based on the difficulty of the questions they answer. This method is similar to adaptive testing systems like the SAT, where each correct or incorrect answer influences the next question. By integrating IRT into AI model evaluation, the researchers aim to create a more balanced and accurate comparison of model performance. Traditionally, the evaluation process for AI models can incur costs equal to or exceeding those of training the models themselves. The Stanford team's solution involves an infrastructure that adaptively selects subsets of questions based on their difficulty, thereby reducing the evaluation costs by 50% or more, and in some cases by over 80%. This system uses AI to analyze and score questions on difficulty, ensuring that the subset of questions used for evaluation is well-calibrated and representative. To build a comprehensive and diverse question bank, the researchers employed AI's generative capabilities. They created a question generator that can produce questions tailored to any desired level of difficulty. This automation helps in continually updating the question bank and filtering out any "contaminated" questions—those that may be biased or irrelevant. The new method, being domain-agnostic, works across various fields, including medicine, mathematics, and law. Koyejo and his team tested their system against 22 datasets and 172 language models, demonstrating its ability to adapt seamlessly to new models and questions. One significant application of their approach was in charting the safety metrics of GPT 3.5 over time. Initially, GPT 3.5 showed improved safety in 2023, but subsequent variations exhibited a regression in performance. Language model safety is crucial, measuring a model's resilience against data manipulation, adversarial attacks, and exploitative behaviors. The introduction of this IRT-based approach marks a substantial improvement in the field. Developers can now achieve better diagnostics and more precise performance evaluations without incurring prohibitive costs. For users, this translates to more reliable and transparent assessments of the AI models they rely on. According to Koyejo, the broader implications extend beyond just technical advancements. "For everyone else, it will mean more rapid progress and greater trust in the quickly evolving tools of artificial intelligence," Koyejo noted. This innovation can accelerate the development of AI by providing a clear and consistent framework for evaluating model improvements, thereby fostering trust and confidence in AI technologies. Industry experts are highly positive about the Stanford researchers' contribution. They see the IRT-based evaluation system as a game-changer that could standardize and streamline the assessment process, making it more accessible to a wider range of organizations. The ability to rapidly and accurately evaluate language models can drive further innovation and adoption of AI, benefiting not only tech companies but also sectors like healthcare, finance, and legal services that increasingly depend on these technologies. Stanford University is renowned for its contributions to AI research, and the Stanford Artificial Intelligence Lab (SAIL) continues to push the boundaries of what is possible with machine learning. This new evaluation approach is a testament to the lab's commitment to advancing AI technology in a way that is both scientifically rigorous and practically viable.