One-sentence Summary

Proposed by Tsinghua University, Ant Digital Technologies, and Ant Group, AgenticDataBench is a comprehensive benchmark for LLM-based data agents that spans 15 real‑world vertical domains with B2B fintech use cases, extracts representative data science skills from Stack Overflow via skill‑aligned hierarchical clustering to quantify coverage and generate realistic tasks for under‑represented domains, and provides fine‑grained skill‑level evaluations of state‑of‑the‑art agents.

Key Contributions

• AgenticDataBench is a comprehensive benchmark with 344 realistic tasks across 15 domains, each annotated with fine‑grained data science skills and covering both real‑world business cases and generated scenarios.
• A hierarchical clustering method with LLM‑based semantic refinement extracts 433 representative data science skills from 6,510 Stack Overflow solutions, enabling quantified coverage and skill‑level benchmark design.
• Evaluation of state‑of‑the‑art data agents on the benchmark yields detailed skill‑level insights, revealing performance patterns and open challenges for autonomous data‑science systems.

Introduction

The authors leverage large language models to automate data science pipelines, yet existing evaluation frameworks for these data agents use only narrow task types and coarse aggregate scores, masking step-level behaviors. Prior benchmarks overlook the complexity of real-world business applications and fail to provide fine-grained, skill-level performance insight. To address this, the authors introduce AgenticDataBench, a comprehensive benchmark grounded in a data-driven skill framework that extracts 433 representative skills from thousands of Stack Overflow solutions through hierarchical clustering with LLM-based semantic refinement. They then select diverse real-world tasks and generate additional tasks with controlled skill coverage, enabling detailed, skill-level analysis of data agent strengths and weaknesses.

Dataset

The authors construct AgenticDataBench, a benchmark designed to evaluate LLM-powered data agents through 344 realistic data science tasks. Each instance includes a task description, a linked dataset, a ground-truth solution, a set of required data science skills, and a task-specific evaluation function.

Dataset composition and sources

• The benchmark combines two complementary subsets:
• Real business tasks from Ant Group’s fintech ecosystem (commercial banking, consumer finance, insurance, automotive, retail, etc.).
• Cross-domain public tasks built on 58 datasets from open repositories (46 Kaggle, 2 UCI ML, 2 Mendeley, and 8 academic/government sources such as UCSD, NYC TLC, U.S. BTS, OWID).

Subset 1: Skill-Diverse Task Selection (102 instances)

• Source: Over 600 anonymized, complex business tasks curated by 30 domain experts (600+ person-hours) from real business operations.
• Filtering and selection:
• Each task is annotated with relevant skills from a hierarchical data science skill tree via LLMs.
• A greedy diversity algorithm selects tasks that maximize coverage of the skill set while respecting a predefined budget, yielding 102 representative tasks.
• Experts refine the selected instances by designing evaluation functions, verifying data privacy, and polishing task descriptions and skill annotations.

Subset 2: Skill Coverage-Driven Task Generation (242 instances)

• Source: 58 public datasets spanning 10 domains not already covered by the business subset, chosen for real-world relevance, inherent complexity (large-scale, heterogeneous formats, noise), and flexible cross-file associations (shared attributes, semantic parallels).
• Generation pipeline:
• Skill graph: Built by merging co-occurrence traces of skills extracted from Stack Overflow solutions and the business task solutions. Node and edge weights reflect real-world skill frequencies and dependencies.
• Skill composition sampling: Sampled paths from the graph generate realistic skill sets that mirror production workflows.
• Ingredient preparation: Real datasets are loaded, skill paths and few-shot task–solution examples (relevant to the sampled skills) are retrieved.
• Task creation: An LLM-based pipeline first profiles each dataset file (basic info, format-specific structure, and cross-file relationships). It then synthesizes a skill-based solution workflow determined by the sampled skills, and finally generates a task description verified against six quality criteria (solvability, skill necessity, conciseness, clarity, actionability, verifiable answer).
• Diversity promotion: Weights of previously used skills and reference examples are dynamically penalized to avoid repetition.
• Expert refinement: 8 experts invest 960 person-hours to verify correctness, adjust data source usage, refine questions, design evaluation functions, and implement solutions. The result is 242 complete instances.

Data usage

• The dataset is used purely as an evaluation benchmark, not for model training. Data agents are given a task description and dataset, and their output is scored by the instance-specific evaluation function (e.g., normalized mean squared error).
• No training/validation/test splits are employed; the full set of 344 instances serves to rank agents and enable fine-grained, skill-level analysis of agent failures.

Data processing and metadata

• Skill hierarchy: All instances are linked to a hierarchical tree of data science skills (e.g., “Handling Missing Data”, “Efficient Data Structures”), obtained by LLM-based extraction, clustering, and recursive abstraction from large-scale solution data, followed by manual curation.
• Data profiling: Public datasets undergo structured profiling that records file schema, column types, missing values, delimiters, and LLM-identified cross-file relationships. This metadata is provided to the task-generation pipeline.
• Skill annotation: Both real and generated tasks carry expert-verified skill labels and skill-usage traces, enabling coverage measurement and diversity control.

Method

The authors introduce a multi-stage pipeline to extract a compact, representative set of data science skills from a large corpus of task solutions. The extracted skill hierarchy is then used to construct benchmark tasks with controlled coverage and divergence. The process consists of four main steps.

Step 1: Vanilla LLM-based Skill Extraction. First, the authors collect 6,510 data science task–solution pairs from Stack Overflow, filtering by relevant tags (e.g., “data-science”, “data-analysis”) and quality signals such as accepted answers or a score above 3. Because many solutions span multiple skills, an LLM is prompted to decompose each solution into stepwise rationales, where each step describes a distinct data science skill while preserving actionable details. The concatenated steps should faithfully reconstruct the original solution. This yields 29,602 fine-grained skill descriptions. However, these raw descriptions suffer from three shortcomings: poor scalability due to sheer volume, high redundancy where many descriptions refer to the same underlying skill, and entanglement where one skill is a high-level abstraction that subsumes another.

Step 2: Embedding-based Skill Clustering. To combat redundancy and entanglement, the authors adopt a hierarchical clustering strategy. Each skill description is embedded with the Qwen3-Embedding model, then UMAP is applied to reduce dimensionality while preserving local manifold structure. Gaussian Mixture Models (GMM) perform soft clustering, allowing a skill to associate with multiple parent clusters. Since vector similarity can be misled by surface-level details, the clusters are later refined with an LLM. To keep clusters manageable for an LLM’s context window, clusters exceeding a predefined size are recursively split via GMM.

Step 3: LLM-based Skill Cluster Refinement. For each cluster, an LLM derives higher-level skill abstractions and groups the original low-level descriptions under them, preserving the parent–child lineage. Next, to merge synonymous skills, the LLM-generated skill descriptions are embedded and clustered with DBSCAN using a strict distance threshold; the shortest description among a synonymous group is chosen as the canonical representation. If the resulting top-level skills are still too numerous, the authors repeat the cluster-and-refine loop. In this recursive pass, they create a summary for each skill by augmenting its LLM description with representative solution steps that have the highest average cosine similarity to other steps using that skill. The summaries are recursively clustered until the number of top-level skills drops below a threshold.

Step 4: Skill Hierarchy Refinement. Entanglement is further addressed by analyzing syntactic token sets in skill descriptions. If a skill’s token set is a subset of another’s, the more general skill is assumed to subsume the specific one and is replaced by its children, updating the hierarchy. Finally, data science experts review the top-level skills to verify proper scoping, diversity, real-world relevance, and suitability for evaluation. This entire pipeline produces 433 well-defined top-level skills that serve as the foundation for benchmark construction.

Experiment

The evaluation compares multiple LLMs paired with four agent harnesses on 344 diverse real-world data science tasks. General-purpose harnesses such as CodeX generally outperform the specialized DA-Agent, but no single harness excels across all domains; LLM-harness compatibility significantly affects performance, and lightweight agents like Smolagents achieve better cost-performance tradeoffs. Skill-level analysis reveals that all agents struggle with heterogeneous and nonrelational data, and failure analysis shows data analysis steps cause the most errors, with domain-specific issues tied to data scale and structure, while simply extending execution budgets does not improve outcomes.

AgenticDataBench includes 155 real-business and 433 public-dataset tasks spanning many domains. Failure analysis reveals that Data Analysis errors dominate, while timeout and self-repair failures point to poor alignment between language models and agent harnesses. Failure patterns diverge substantially across domains, shaped by data scale, file formats, and task complexity. Data Analysis errors are the most frequent failure category despite not being the most called skill. Global Limit Exceeded, Single-Step Timeout, and Self-Repair Failure together indicate weak LLM–harness integration. Marketing tasks suffer the largest share of timeout failures due to repeated loading of large (≈1 GB) files. Healthcare shows the highest Self-Repair Failure rate, as heterogeneous formats like ARFF trigger parsing errors. Loan Model tasks are most impacted by Data Modeling failures, driven by complex feature engineering on wide tables. Increasing time budgets does not improve scores; it extends unproductive loops or misleads reasoning.

AgenticDataBench expands the scope of data agent evaluation by covering 433 data science skills and incorporating real business data, addressing the narrow task types and coarse labels of prior benchmarks. It features tasks with much longer solution code and larger data volumes, reflecting the complexity of real-world data science workflows. Existing benchmarks provide only a handful of task tags and limited data modalities, while AgenticDataBench delivers fine-grained, skill-level annotations and supports modalities including scripts and databases. AgenticDataBench covers 433 distinct skills, far exceeding the skill coverage of any existing benchmark, and uniquely combines real business data with public datasets. Tasks in AgenticDataBench average 113.6 lines of solution code and 493.4 MB of data per task, substantially longer and larger than those in prior benchmarks, indicating greater task complexity.

Domains vary widely in data scale and structure: Marketing has extremely large per-task files (3.3 GB), Loan Model comprises many small files (121 total) with high per-task file counts, and Healthcare includes heterogeneous formats. These characteristics directly drive distinct agent failure patterns, with oversized files causing timeouts, format diversity inducing self-repair errors, and complex feature engineering leading to modeling failures. Marketing’s multi-gigabyte per-task payload leads to the highest share of global limit exceeded and single-step timeout failures. Healthcare’s heterogeneous data formats (e.g., ARFF) cause the highest self-repair failure rate from frequent parsing errors. Loan Model’s combination of many small files per task and reliance on complex feature derivation makes it most susceptible to data modeling failures.

Across the four domains, the most frequent skills (by TF-IDF) do not align with the most challenging ones (lowest LLM-assigned scores). Representative skills include metadata review, dataframe management, and model training, while challenging skills such as SQL optimization, data transformation, statistical testing, and performance metrics often involve cross‑stage or analytical operations. These challenging skills correspond to categories where data agents show high failure rates, notably Data Analysis and Data Manipulation. In the financial domain, metadata review and query construction are frequent, but SQL optimization and data transformation are the hardest. For loan model tasks, dataframe column management and model training are common, while data comparison, validation, and statistical testing prove most difficult. Loan risk analysis frequently uses custom value replacement and helper functions, yet data preprocessing and normalization are the most challenging skills. Marketing tasks most often involve model training, but performance metrics and optimization are the lowest‑scoring skills.

Across AgenticDataBench, production-grade agent harnesses like CodeX and Smolagents outperform the lightweight data-science-specific DA-Agent. The best-performing large language model varies by harness: Claude Sonnet 4.6 leads within DA-Agent and Claude Code, Qwen3.5 within Smolagents, and Kimi-K2.5 within CodeX. Domain-specific strengths emerge, with Smolagents excelling on Marketing tasks that involve large single files. CodeX paired with Kimi-K2.5, Smolagents with Qwen3.5, and Smolagents with Claude 4.6 are the three highest-scoring agent–LLM combinations overall. Smolagents attains the best Marketing performance, where many tasks handle single data files around 1 GB or larger. Claude 4.6's strong coding abilities make it the top LLM in DA-Agent and Claude Code, while Qwen3.5's better adaptation to Smolagents' prompts gives it the lead within that harness.

AgenticDataBench introduces a broad data-science benchmark spanning 433 skills with real business and large-scale public data, revealing that data analysis errors dominate failures and that domain characteristics like file size, format heterogeneity, and feature complexity drive distinct breakdown patterns. Poor alignment between language models and agent harnesses leads to timeout and self-repair failures, and longer time budgets fail to improve outcomes. Among agent harnesses, production-grade systems such as CodeX and Smolagents outperform a lightweight counterpart, with Smolagents excelling on large-file marketing tasks and the best-performing LLM varying by harness.