Google is testing a new AI-powered search tool called Scholar Labs, designed to help researchers find relevant scientific studies by understanding the meaning and context behind complex queries. The tool uses artificial intelligence to analyze the full text of research papers, their authors, publication venues, and citation patterns to surface results that are most useful for a user’s specific research question. However, the approach raises questions about how scientists will assess the quality of the studies it surfaces. In a demonstration, Scholar Labs was asked to find research on brain-computer interfaces (BCIs) for stroke patients. The top result was a 2024 review paper published in Applied Sciences, a journal with an impact factor of 2.5—far lower than prestigious journals like Nature, which has an impact factor of 48.5. The tool explained why the paper matched the query, noting its focus on noninvasive electroencephalogram signals and leading algorithms in the field. But unlike traditional search tools, Scholar Labs does not let users filter results by citation count or journal impact factor—two widely used metrics in academia to gauge a paper’s influence and credibility. Google’s spokesperson Lisa Oguike explained that these metrics can be misleading because they vary significantly across scientific fields and may overlook recent or interdisciplinary work. Limiting results to only high-impact journals or highly cited papers, she said, could cause researchers to miss important studies, especially in emerging or cross-disciplinary areas. While citation counts and impact factors are imperfect, they are still widely used as quick indicators of a paper’s reach and acceptance. Professor James Smoliga of Tufts University, a regular user of Google Scholar, admitted he still defaults to citing highly cited papers, even when he knows some of them have flawed methods. “I know it’s not the right way, but what else can I do?” he said. Matthew Schrag, an associate professor of neurology at Vanderbilt University Medical Center, agrees that these metrics are coarse. They reflect a paper’s visibility and social traction more than its scientific rigor. He noted that the scientific community has become more vigilant in recent years, with researchers uncovering data manipulation, image fraud, and other issues in high-profile journals. Still, he acknowledged that without clear benchmarks, scientists often rely on citation counts and journal reputation as proxies for quality. Schrag believes AI tools like Scholar Labs have potential. They could help surface overlooked or niche research, especially in interdisciplinary fields, and offer deeper context—such as how a paper is discussed on social media or in other scholarly communities. But he stressed that scientists must remain the final judges of quality. “You have to read the papers, understand the methods, and engage critically,” he said. “Algorithms shouldn’t be the final arbiters of what counts as good science.” Google says Scholar Labs is still in early testing and plans to incorporate user feedback. Access is currently limited to a waitlist. While the tool represents a shift toward more semantic, context-aware search, it also underscores a growing challenge: how to balance AI’s ability to find relevant information with the need for human judgment in evaluating scientific quality.