Command Palette
Search for a command to run...
Dataset Compilation | AI Agent Evaluation Datasets: 10 Datasets Released by Microsoft, Peking University, HKU, Shanghai Jiao Tong University, etc., Covering Everything From long-range Memory to real-world Task execution.
With the continuous expansion of large-scale model capabilities, AI agents are rapidly evolving from "dialogue tools" to "task executors," and are beginning to be widely used in scenarios such as automated office work, code generation, data analysis, and complex process handling. Compared to traditional question-answering large language models (LLMs), the core change in agents lies in:It no longer just answers questions, but is able to break down tasks, call upon tools, and autonomously advance until the goal is achieved.
As the capabilities of agents expand, the datasets used for training and evaluation become increasingly crucial. Unlike traditional single-round question-answering or static task data, agent-related datasets emphasize "process capabilities," such as long-term planning, multi-step reasoning, tool usage, and memory. These data determine whether the model can run stably in real-world, complex tasks, and directly affect the reliability and upper limit of the agent system.
This article compiles 10 relevant datasets.It covers several core capability areas in current agent research: long context understanding and memory assessment, complex task planning and multi-step reasoning, tool invocation and interactive execution capabilities, and task completion capabilities in real or simulated environments.These data collectively constitute a systematic characterization of the capabilities of AI Agents.
The composition of these datasets also reveals a shift in the evaluation and training data themselves: from static question-and-answer annotations to modeling the interaction process and behavioral trajectories, emphasizing the data's support for the model's ability to "do things" rather than just its ability to "answer".
The following datasets are available for use online at HyperAI.It aims to accelerate research and exploration by researchers and developers in related fields.
More high-quality datasets:
Dataset Recommendation
1. RHELM Long-Term Memory Assessment Dataset
* Use online:
RHELM is a long-term memory assessment dataset released by Microsoft in 2026. The related paper is titled "Beyond Static Dialogues: Benchmarking Realistic, Heterogeneous, and Evolving Long-Term Memory". It aims to improve the long-term memory, multi-hop reasoning, and temporal information synthesis capabilities of large models in complex dynamic scenarios. It is widely used in research scenarios such as long-term memory evaluation of large language models, verification of long-term interaction capabilities of AI assistants, multi-hop reasoning of large models, temporal information fusion, and hallucination detection.
The dataset contains 10 sets of virtual character profiles, 1,305 question-answer pairs, 629 JSON-formatted conversations, 625 TXT-formatted email threads, and 1,053 MD and HTML-formatted attachment documents. The accompanying questions cover seven core types: attachment referencing, mixed reasoning, fact-finding, illusion detection, information aggregation, time-series analysis, and misleading questions.
2. MemLens Multimodal Long Context Benchmark Dataset
* Use online:
MemLens is a benchmark dataset for evaluating long-range dialogue memory in visual language models. It tests the model's ability to retrieve, recall, update, and infer visual and textual information embedded in multi-conversation dialogues within context windows of 32K, 64K, 128K, and 256K. The dataset contains 789 items covering five evaluation types: information retrieval, knowledge updating, temporal reasoning, multi-conversation reasoning, and rejection (Abstention), and provides four context length configurations (32K/64K/128K/256K).
The dataset also provides a fixed stratified sample subset of 195 questions specifically for evaluating memory-augmented agents to balance inference costs.
3. LongBlocks Long Context Multilingual Question Answering Dataset
* Use online:
LongBlocks is a multilingual synthetic dataset with long context, released in 2026 by the University of Lisbon, the Institute of Telecomunicações, TransPerfect, and other institutions. This dataset contains approximately 194,000 long-context question-and-answer examples, covering long document corpora such as books, web page text, Wikipedia entries, arXiv papers, programming code, and community Q&A.
4. AgentTrove intelligent agent interaction trajectory dataset
* Use online:
AgentTrove is a large-scale open-source dataset of agent interaction trajectories released by the OpenThoughts-Agent team. This dataset contains 1,696,847 rows of data, sourced from 219 datasets, covering tasks such as code repair, shell scripting, mathematical problem solving, programming competitions, and general computing usage. All trajectories were collected using the open-source Harbor agent evaluation and data generation framework and are uniformly published in the Terminus-2 harness format (a ShareGPT-like dialogue layout).
5. Claw-Eval Real-World Benchmark Dataset
* Use online:
Claw-Eval is an end-to-end transparent evaluation benchmark dataset for evaluating AI agents in real-world tasks, released in 2026 by Peking University and the University of Hong Kong. The related paper is titled Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents. It aims to evaluate the ability of autonomous agents to perform tasks, invoke tools, understand multimodalities, and interact in real-world environments. It is widely used in agent system evaluation, automated task execution, multimodal agent research, and large model capability analysis.
This dataset supports both English and Chinese languages and includes three core task groups: General, Multimodal, and Multi-turn, covering a total of 24 task categories such as communication, finance, office, and productivity tools.
6. OpenMementos Context Memory Compressed Dataset
* Use online:
OpenMementos is a context-memory compression dataset released by Microsoft in 2026, designed for modeling long-chain inference and context management capabilities of large models. This dataset aims to train models to perform context compression and continuous inference, thereby supporting complex multi-step inference tasks within a limited context window. It is widely applicable to research scenarios such as long-chain inference modeling, memory-enhanced model training, and efficient generation.
This dataset is built on the OpenThoughts inference dataset and contains 228,557 structured inference tracks, including 123,333 math tracks, 61,485 science tracks, and 43,739 programming tracks. The average number of sentences per track is 187.
7. MIA Multi-Step Inference and Decision Trajectory Dataset
* Use online:
MIA (Memory Intelligence Agent) is a dataset jointly released in April 2026 by East China Normal University, Shanghai Innovation Institute, and Harbin Institute of Technology. It is used to train and evaluate intelligent agents with long-term memory and task execution capabilities. The related research paper is titled "Memory Intelligence Agent," aiming to improve the long-term memory utilization and multi-step decision-making abilities of intelligent agents. This dataset contains approximately 21,000 reasoning trajectory data, covering the entire process of problem solving, planning, searching, and execution, and is suitable for agent reasoning and reinforcement learning research.
8. ToolACE Complex Tool Learning Dialogue Dataset
* Use online:
ToolACE is an automated agent pipeline dataset for tool learning tasks, released in 2024 by Shanghai Jiao Tong University in collaboration with the University of Science and Technology of China, Huawei Noah's Ark Lab, and other institutions. The related paper is titled ToolACE: Winning the Points of LLM Function Calling. It aims to generate accurate, complex, and diverse tool learning data, especially addressing real-world problems such as insufficient data quality and limited scenarios in tool learning.
This dataset contains multi-step conversation examples, calling a total of 26,507 diverse APIs. The samples are generated through multi-agent interactions and undergo a two-layer quality assurance process of rule checking and model validation. Each dialogue represents a multi-step, multi-source information retrieval and analysis task, realistically simulating tool call scenarios and providing high-value training data for LLM (Low-Level Modeling).
9. Creative Professionals Creative Task Instruction Dataset
* Use online:
Creative Professionals Agentic Tasks is a large-scale, high-fidelity synthetic task dataset designed for the training, evaluation, and fine-tuning of multimodal AI agents. It contains 1,070,917 agent command operations, covering 36 creative, technical, and engineering software environments. The dataset aims to explore complex software interactions and multi-step reasoning.
10. AgentNet Desktop Operation Task Dataset
* Use online:
AgentNet is the first large-scale desktop computer agent trajectory dataset released in 2025 by the XLANG Lab at the University of Hong Kong in collaboration with Moonshot AI, Stanford University, and other institutions. The related paper is titled "OPENCUA: Open Foundations for Computer-Use Agents". It aims to support and evaluate cross-platform GUI-operated agents and vision-language-action (VLA) models.
This dataset contains 22.6K manually annotated computer usage task traces, covering Windows, macOS, and Ubuntu, and over 200 applications and websites. The scenarios fall into four categories: office, professional, daily, and system. It is suitable for training and evaluating desktop automation, multi-application processes, and cross-platform agents.
These are all the datasets recommended in this issue. Download and use them with one click!
