One-sentence Summary

Researchers from Zhejiang University, Microsoft Research Asia, and Fudan University introduce TASKBENCH, a comprehensive framework that evaluates large language models for task automation across the three critical stages of task decomposition, tool selection, and parameter prediction, leveraging a Tool Graph representation and back-instruct method for data generation, and employing the TASKEVAL methodology that combines automated construction with rigorous human verification to reliably assess autonomous agent capabilities.

Key Contributions

• The paper introduces TASKBENCH, a scalable and reliable benchmark framework that evaluates LLM capabilities in task automation across three critical stages: task decomposition, tool selection, and parameter prediction.
• The work presents the Tool Graph concept, which models real-world dependencies and interconnections between tools, together with a Back-Instruct method that aligns tool subgraphs with user instructions to reduce hallucination and improve data authenticity.
• The paper proposes TASKEVAL, a multi-faceted evaluation methodology combining automated construction with rigorous human verification, and experimental results demonstrate that TASKBENCH effectively reflects LLM performance across task complexities and domains while showing high correlations with human evaluation.

Introduction

Large language model empowered autonomous agents have become a prominent direction in AI research, with task automation as their core capability. This involves using LLMs to interpret user instructions and autonomously execute objectives through multiple stages: task decomposition, tool selection, and parameter prediction. However, the field lacks a systematic and standardized benchmark to evaluate these abilities thoroughly. Real-world tasks often include intricate instructions, multiple subtasks, and interdependent tools, which makes modeling tool dependencies a key challenge in both dataset construction and evaluation metrics.

Existing benchmarks fall short. Traditional benchmarks such as MMLU, GSM8K, and AGIEval assess general knowledge or reasoning, not task automation. Tool-augmented benchmarks like APIBench, ToolBench, and MetaTool focus on specific aspects such as tool selection or question answering, but they rarely cover all stages of task automation or adequately model complex tool dependencies. Template-based data generation and heavy reliance on human verification in these approaches further limit scalability and diversity.

To address these gaps, the authors introduce TASKBENCH, a comprehensive benchmark for evaluating LLMs in task automation. They first propose the concept of a Tool Graph, which represents relationships and dependencies among tools, overcoming the limitations of simple API documentation or template-based approaches. Using this graph, they apply a back-instruct strategy to generate diverse and realistic user instructions from sampled subgraphs. A multi-step quality control pipeline, including structured sampling, LLM-based and rule-based self-critic mechanisms, and extensive human verification, ensures data naturalness and utility. The authors also present TASKEVAL, a multi-faceted evaluation framework that assesses LLM performance across task decomposition, tool selection, and parameter prediction. Experimental results show that TASKBENCH, supported by TASKEVAL, effectively reflects LLM capabilities across multiple dimensions and correlates strongly with human evaluation.

Dataset

The authors introduce TASKBENCH, a benchmark designed for evaluating LLMs in task automation. Unlike prior approaches that rely on collection or instruction methods, the authors use a back-instruct strategy to simulate data that reflects the complex relationships among multiple tasks and more practical user instructions.

• Tool Graph construction: The dataset is built around a Tool Graph (TG), formally defined as G = (T, D), where T is a collection of tools and D is a set of dependencies. The authors focus on two dependency types: resource dependencies (when the input type of tool a matches the output type of tool b) and temporal dependencies (which define the execution order between tools). Tool graphs are built from three domains:

  • Hugging Face: AI models treated as tools, with connections determined by input-output type matching. This graph has 23 tools and 225 edges.
  • Multimedia: Broader user-centric tools such as file downloaders and video editors, using the same connection policy. This graph has 40 nodes and 449 edges.
  • Daily Life APIs: Services such as web search and shopping, connected primarily by temporal dependencies. This graph has 40 nodes and 1,560 edges.

• Sub-graph sampling: From the tool graph, the authors sample sub-graphs that capture valid tool invocation patterns. Three substructure types are used:

  • Node, representing a single standalone tool invocation.
  • Chain, representing sequential tool invocations.
  • DAG, representing more complex invocations where a tool depends on or influences multiple tools. The sampling function takes a mode and a size (ranging from 1 to 10 tools) to control the topology and magnitude of the sampled sub-graph.

• Back-instruct data generation: The authors use GPT-4 as the data engine to convert sampled sub-graphs into user instructions. BackInstruct1 generates a user instruction from the sampled tool skeleton, while BackInstruct2 fills in tool execution parameters and produces the task decomposition steps along with the final tool invocation graph.

• Quality control: To ensure data quality, the authors apply multi-step filtering:

  • Self-critic mechanisms, with an LLM-based variant and a rule-based variant, that check alignment between generated instructions and the sampled tool sub-graphs.
  • Human verification, where experts review instructions for naturalness, complexity, and alignment after a calibration session to standardize criteria.

• Dataset statistics and usage: The dataset covers three domains and uses a mode ratio of 3:7:8 for Node:Chain:DAG sampling. The distribution for the number of tools is set as {0.1, 0.2, 0.3, 0.2, 0.1, 0.05, 0.025, 0.025, 0.025}. The benchmark enables LLMs to complete the full task automation process, including task decomposition, tool selection, and parameter prediction.

Method

The authors introduce a data generation framework utilizing a "Back-Instruct" strategy to simulate user instructions for task automation. This methodology enables the evaluation of LLMs on task decomposition, tool selection, and parameter prediction. The complete workflow is depicted in the framework diagram below.

The pipeline initiates with Tool Graph Construction. The authors aggregate standalone tools from diverse sources, including Hugging Face for AI tasks, Multimedia tools for user-centric operations, and Daily Life APIs for services like web search. These tools are organized into a graph $\mathcal{G}$G by establishing edges based on dependencies. Two dependency types are prioritized: resource dependencies, where the input type of one tool matches the output type of another, and temporal dependencies, which enforce a sequential order between tools.

Following graph construction, the system executes Graph Sampling. Subgraphs are extracted from $\mathcal{G}$G to create varied structural templates, specifically single nodes, linear chains, and Directed Acyclic Graphs (DAGs). The sampling process is calibrated with specific ratios to balance the complexity and the number of tools in the generated data.

The central mechanism is Back-Instruct, which synthesizes user instructions inversely from the sampled subgraph $\mathcal{G}_s$Gs​. First, an LLM generates a user instruction $I$I that necessitates the tools within the subgraph, defined as $\text{BackInstruct}_1(\mathcal{G}_s) \to I$BackInstruct1​(Gs​)→I. Because the subgraph only outlines the tool invocation skeleton, a subsequent step fills in the missing execution details. The LLM populates the parameters to produce the final tool invocation graph $\bar{\mathcal{G}}$Gˉ​ and the associated task decomposition steps $P$P, formulated as $\text{BackInstruct}_2(\mathcal{G}_s, I) \to \{P, \bar{\mathcal{G}}\}$BackInstruct2​(Gs​,I)→{P,Gˉ​}.

To maintain high data quality, the framework incorporates a rigorous Quality Control Mechanism. This includes a Self-Critic component with two variants: an LLM-based critic that assesses the alignment between generated instructions and sampled subgraphs, and a rule-based critic that verifies consistency by comparing nodes and edges. Additionally, a Human Verification process is implemented where experts review the data for logical coherence, naturalness, and task alignment, ensuring the final dataset is robust for benchmarking.

Experiment

The evaluation validates TaskBench dataset quality via human assessments of Naturalness, Complexity, and Alignment, showing Back-Instruct with graph edges yields more natural and complex instructions than its no-edge variant or Self-Instruct. Across TaskEval, which assesses task decomposition, tool selection, and parameter prediction, GPT series models consistently lead, with edge prediction proving harder than node prediction and open-source models struggling on complex dependencies. Key contributing factors include code pre-training, instruction following, and RLHF alignment, and TaskEval's strong correlation with human evaluation confirms its reliability as a benchmark.

TaskBench distinguishes itself from existing tool-augmented LLM benchmarks through its use of a Tool Graph for realistic dependency modeling and a Back-Instruct strategy for data generation, which improves authenticity and reduces hallucination risk. Unlike prior benchmarks that rely heavily on template-based generation or API documentation alone, TaskBench offers more comprehensive quality control and evaluation across task decomposition, tool selection, and parameter prediction, with a larger and more complex dataset spanning single tools to multi-tool graph structures. TaskBench uniquely incorporates a Tool Graph to model real-world tool dependencies, while most other benchmarks do not support such dependency modeling. Data generation in TaskBench combines Tool Graph with Back-Instruct, contrasting with the API documentation or template-based methods used by other benchmarks. Its evaluation covers task decomposition, tool selection, and parameter prediction, while many benchmarks focus only on tool selection or API correctness. TaskBench supports tool complexity ranging from a single tool to complex tool graph structures, similar to some others but with a more integrated dependency approach. The quality control for TaskBench uses LLM self-critique plus rule-based verification, whereas other benchmarks rely primarily on human verification.

Human evaluators rated Back-Instruct higher than its ablated version and Self-Instruct across all metrics, with notable gains in naturalness and complexity. Removing edge information from Back-Instruct substantially reduced naturalness and complexity scores, while Self-Instruct scored lowest overall despite comparable alignment. Back-Instruct outperforms both the no-edges variant and Self-Instruct on naturalness, complexity, and overall quality. Dropping edge information mainly hurts naturalness and complexity, while alignment stays relatively stable. Self-Instruct produces much lower naturalness and complexity scores, indicating weaker instruction realism and task intricacy.

The evaluation of tool selection shows that LLMs consistently perform better at identifying individual tools (node prediction) than at capturing dependencies between them (edge prediction), with edge F1 scores often lagging by about 20 points. Model rankings vary with structure complexity: open-source models like Mistral-7b and CodeLlama-13b compete with GPT-3.5-Turbo on simpler node structures but fall behind on more complex dependency graphs, while GPT-4 maintains the most stable and highest performance across all structure types. Edge prediction is consistently harder than node prediction across all models, with F1 gaps around 20 points. Open-source models can match GPT-3.5-Turbo on simple node tasks but struggle with complex dependencies. GPT-4 leads overall and shows the most consistent performance across chain and DAG structures.

The evaluation of tool parameter prediction shows that GPT-4 consistently achieves the highest scores across all workflow types, with particularly strong performance in both parameter name and value identification. Closed-source models like Gemini-Pro and Claude-2 trail GPT-4 but outperform open-source alternatives, which generally show lower value-level F1 scores, indicating a gap in precise parameter assignment. GPT-4 leads all models on both t-F1 and v-F1 metrics across Node, Chain, DAG, and Overall settings. Open-source models such as Mistral-7B show the weakest performance, especially on v-F1, highlighting challenges in predicting correct parameter values. Across all models, v-F1 scores are lower than t-F1 scores, suggesting that assigning correct parameter values is more difficult than identifying parameter names.

The experiments center on TaskBench, a tool-augmented LLM benchmark using a Tool Graph for dependency modeling and Back-Instruct for data generation, which human evaluators found more natural and complex than ablated or Self-Instruct variants. Across tasks, edge prediction proved harder than node prediction for tool selection, with GPT-4 leading consistently while open-source models lag on complex dependencies. For parameter prediction, all models found value identification harder than name identification, with GPT-4 outperforming all others, followed by closed-source models and then open-source ones. Overall, the findings validate TaskBench's design and reveal that dependency modeling and precise parameter assignment remain key challenges for tool-augmented LLMs.