Argonne National Laboratory in the United States Proposed ChemGraph, Which Uses 13 Benchmark Tests to Evaluate the Value of Agents in the Field of Computational chemistry.

In recent years, the rapid development of artificial intelligence, especially the progress of large language models (LLM), has opened up new possibilities for automated scientific research.**Several LLM-based agents have already been developed to assist users in completing various chemistry-related tasks.**For example, Bran et al. developed ChemCrow, a chemical agent driven by a large language model, which can perform chemical synthesis tasks for a variety of molecules; McNaughton et al. developed CACTUS, a large language model agent that integrates cheminformatics tools, which can help researchers complete tasks such as molecular property prediction, similarity search, and drug similarity assessment; White and colleagues proposed MDCrow, an LLM agent assistant that can perform molecular dynamics (MD) workflows.
In this context,**A research team from Argonne National Laboratory in the United States has proposed ChemGraph, an intelligent agent system driven by a large language model.**It is designed to perform molecular simulation workflows in the field of computational chemistry.
Researchers evaluated ChemGraph on 13 benchmark tasks. The results showed that for simple tasks requiring only a few tools, smaller, large language models (such as GPT-4o-mini and Claude-3.5-haiku) achieved high accuracy and stability. However, as task complexity increased, the performance of these models significantly decreased, while the large model (GPT-4o) maintained strong performance. By strategically breaking down complex tasks into smaller, more manageable subtasks, even with smaller models, ChemGraph's performance could be improved to match, and in some cases even surpass, that of GPT-4o.
The related research findings, titled "ChemGraph as an agentic framework for computational chemistry workflows," have been published in the Nature sub-journal Communications Chemistry.

Research highlights:

ChemGraph leverages graph neural network-based models to achieve accurate and efficient computation, while combining LLM's natural language understanding, task planning, and scientific reasoning capabilities, providing an intuitive and interactive user interface.

ChemGraph can perform a range of tasks, from generating SMILES strings and molecular structures to geometry optimization, vibrational analysis, and thermochemical calculations.

ChemGraph supports users in using a variety of molecular simulation methods, including semi-empirical methods, machine learning potential functions, and density functional theory (DFT).

Paper address:

https://www.nature.com/articles/s42004-025-01776-9

Thirteen benchmark experiments establish an evaluation system for computational chemistry intelligent agents.

**Although some benchmark frameworks for agents already exist, there is still no unified benchmark for agent evaluation in the field of computational chemistry. Therefore, the researchers designed 13 benchmark experiments to evaluate the capabilities of ChemGraph.**These experiments aim to test ChemGraph's ability to accomplish tasks using six integrated tools, including single tool calls and toolchain calls.
Based on the different types of user input, these tasks were divided into three categories: (1) molecular name; (2) SMILES string; (3) chemical reaction. The table below summarizes 360 independent evaluations conducted in 13 experiments.

In the first 11 experiments, the tasks mainly revolved around molecular names or SMILES strings, with each task requiring a maximum of 4 tool calls or 4 sub-tasks. In contrast, the last two experiments focused on calculating the thermochemical properties of chemical reactions, significantly increasing the complexity and requiring 9 to 12 tool calls depending on the number of reactants and products. These tasks required performing thermochemical calculations for each substance separately and further constructing reaction-level properties based on the previously obtained results.

ChemGraph, an intelligent agent framework driven by a large language model

ChemGraph is an intelligent agent framework driven by a large language model, designed to automate molecular simulation workflows through structured tool calls and reasoning capabilities. It is based on LangGraph and follows the ReAct framework.

LangGraph introduces a graph-based execution model designed for more robust multi-agent collaboration. A typical LangGraph workflow consists of three parts:

• StateState is a shared data structure that represents the current state of a system.
• NodesA node is a Python function that defines the logic of an agent. It takes the current state as input and returns the updated state. A node can be a large language model or a function that performs a specific operation.
• EdgesAn edge is a function that controls the flow of messages, determining which node to execute next. Edges can be divided into directed edges or conditional edges—directed edges represent a fixed, unidirectional transmission of messages from one node to another; in contrast, conditional edges offer greater flexibility, allowing messages to flow to different nodes based on specific conditions.

The diagram below illustrates the general structure of ChemGraph: First, the large language model agent receives a set of predefined tools. Based on the user's prompts, the agent decides which tool to invoke. After each tool invocation, the large language model receives the result and determines whether another tool invocation is needed. Once all tool invocations are complete, the message can continue to be delivered along one of two paths.

In the first path, ChemGraph, similar to a traditional large language model, returns a human-like response containing the results of tool calls. In the second mode, the message is forwarded to a second large language model agent, which generates structured output (in a predefined JSON format) directly in response to the user's request. This dual-mode design supports both natural language interaction and systematic evaluation, enabling ChemGraph to meet both typical use cases and evaluation workflows.

The following diagram further illustrates how ChemGraph invokes tools and coordinates their output to accomplish computational chemistry tasks:

In this example, the user requests ChemGraph to calculate the enthalpy of the methane combustion reaction at 400 K using the GFN2-xTB method. ChemGraph (equipped with GPT-4o-mini) first converts the chemical name of each molecule in the reaction into a SMILES string (tool calls 1 through 4). Then, based on these SMILES strings, it generates atomic coordinates in an AtomsData data structure (tool calls 5 through 8). Finally, ChemGraph performs thermodynamic calculations using the generated atomic coordinates and user-specified parameters (such as calculator type, temperature, etc.).

Multi-agent architecture can significantly improve model performance

The research team systematically evaluated ChemGraph's performance on computational chemistry tasks of varying complexity through 13 benchmark tests:

Single agent evaluation

First, the researchers evaluated the performance of the single-agent ChemGraph on 13 experimental tasks using three large language models—GPT-4o-mini, Claude-3.5-haiku, and Qwen-2.5-14B. For GPT-4o, only its performance on the last two problems (react2enthalpy and react2gibbs) was evaluated. The following figure shows the accuracy of different models on different tasks:

existname2smi taskmiddle—Claude-3.5-haiku did not consistently invoke the molecule_name_to_smiles tool. Instead, it sometimes attempted to manually construct the SMILES string using its internal chemical knowledge, or refused to use the tool altogether. As a result, Claude-3.5-haiku had the lowest average number of tool calls per molecule among the three models in this task. GPT-4o-mini was able to accurately complete the experiment and generate the correct answer. Qwen-2.5-14B was generally able to generate accurate tool calls, including the correct tool names and parameters; however, its errors mainly occurred during the extraction of results from the tool output.

existThe name2xyz, name2opt, and name2vib tasksmiddle—The requirement is that large language models not only accurately invoke tools, but also generate correct and structured output results. GPT-4o-mini and Claude-3.5-haiku both performed well in this regard, achieving accuracy exceeding 83%.

existsmi2xyz to smi2file taskmiddle—Claude-3.5-haiku and GPT-4o-mini maintained strong and stable performance, with both achieving an accuracy of over 83%; Qwen-2.5-14B also showed improved performance in this set of tasks, with a minimum accuracy of 77%.

existreact2enthalpy and react2gibbsTask****middle—Qwen-2.5-14B performed the weakest, with an accuracy below 20%; GPT-4o-mini achieved moderate results, with accuracies of 40% and 49% respectively; Claude-3.5-haiku outperformed other small-scale large language models, achieving accuracies of 67% and 69% respectively; GPT-4o achieved the highest performance, with an average accuracy exceeding 83% in both tasks.

Multi-agent evaluation

The researchers further evaluated the performance of the multi-agent ChemGraph using GPT-4o-mini, Claude-3.5-haiku, Qwen-2.5-14B, and GPT-4o (see figure below) and compared the results with the performance of the single agent.

The results show that**Multi-agent systems significantly improve the accuracy of different models.**In the react2enthalpy task, GPT-4o-mini improved its accuracy from 401 TP3T to 871 TP3T, and Claude-3.5-haiku improved from 671 TP3T to 871 TP3T, both results being the average of three independent runs. Notably, both models outperformed the single-agent GPT-4o's baseline performance of 831 TP3T. A similar trend was observed in the final task, react2gibbs: GPT-4o-mini's accuracy improved from 491 TP3T to 871 TP3T, and Claude-3.5-haiku improved from 691 TP3T to 931 TP3T. In contrast, Qwen-2.5-14B only achieved limited improvement, mainly due to its frequent tool invocation errors, which limited the aggregate agent's ability to generate accurate answers.

Ultimately, in multi-agent mode, GPT-4o achieved perfect accuracy, completing two tasks (react2enthalpy and react2gibbs) with a success rate of 100%.

Final Thoughts

ChemGraph, a large language model intelligent agent system for computational chemistry and materials science, demonstrates the potential of AI to automate scientific research workflows. The research team stated that they will further expand ChemGraph's application capabilities in large-scale simulation tasks by integrating more tools, optimizing the agent architecture, and enhancing high-performance computing support. Simultaneously, the system utilizes Docker containers to ensure operational security and continuously improves the reliability of the agent.

It's important to note that ChemGraph is not a replacement for traditional computational chemistry software, but rather serves as an intelligent collaboration layer connecting researchers and simulation tools through natural language interaction. With the development of large open-source models, ChemGraph is also expected to reduce the cost of AI research applications and enable intelligent scientific agents to play a greater role in fields such as materials discovery and molecular design.

References:

https://phys.org/news/2026\-07\-ai\-framework\-battery\-combustion\-materials.html

https://www.nature.com/articles/s42004\-025\-01776\-9