AgentWorldBench is a benchmark dataset for comprehensive evaluation of language world models, released by Qwen in 2026. The related research paper is... Qwen-agentworld: language world models for general agentsIt aims to evaluate the environmental modeling and reasoning capabilities of language world models, and is widely used in large model agent capability evaluation, tool invocation and system interaction verification, as well as automated software engineering and operating system-level task research. This dataset contains 2,170 samples with an average of 22.8 rounds of interaction. It is built based on real-world model trajectories from mainstream benchmarks such as Tool Decathlon, Terminal-Bench 1.0/2.0, and OSWorld-Verified. Each evaluation sample is accompanied by a standard answer obtained through execution in a real environment. The predicted environmental observations are scored across five dimensions: format, factuality, consistency, realism, and quality, in order to explore the reasoning, knowledge, and long-contextuality capabilities required for environmental simulation.

Dataset composition

• MCP: 286 samples, with an average of 23.1 rounds of interaction, covering API server responses, tool call results, database status, and service protocols.
• Search: 458 samples, average 15.5 interactions, covering search engine results, URLs, summaries, rankings, and page content.
• Terminal: 354 samples, with an average of 26.7 rounds of interaction, covering command-line environment, shell output, file system status, and process behavior.
• SWE: 472 samples, average 28.1 rounds of interaction, covering IDE/code editing environment, git diff, test results and compilation errors.
• Android: 200 samples, average interaction time 37.8 rounds, covering changes in the Android UI hierarchy after touch/gesture operations.
• Web: 200 samples, average 14.2 rounds of interaction, covering changes in the browser DOM state after user interaction.
• OS: 200 samples, averaging 12.7 rounds, covering desktop operating system status, file system, window management, and application behavior.

Citation

@article{zuo2026qwen,
title={Qwen-agentworld: language world models for general agents},
author={Zuo, Yuxin and Xiao, Zikai and Sheng, Li and Huang, Fei and Tu, Jianhong and Liu, Yuxuan and Tang, Tianyi and Hu, Xiaomeng and Su, Yang and Lan, Qingfeng and others},
journal={arXiv preprint arXiv:2606.24597},
year={2026}
}