One-sentence Summary

Researchers from HKUST(GZ) and Tsinghua University introduce DataSpace, a benchmark of 410 cross-language tasks over heterogeneous workspaces where data agents produce verifiable tabular results, built with DataSpace-Builder and a deterministic evaluator, which served as the official KDD Cup 2026 benchmark and achieved a top accuracy of 66.34%, underscoring challenges in multimodal evidence integration.

Key Contributions

• The paper introduces DataSpace, a benchmark of 410 cross-language tasks and 7,439 artifacts across CSV, JSON, SQLite, Markdown, PDF, and video, designed for verifiable tabular-output analytics. It served as the KDD Cup 2026 evaluation benchmark, and experiments across six frontier multimodal models and five agent harnesses show a best accuracy of 66.34%, with multimodal evidence and joins consistently reducing performance, leaving the benchmark unsaturated.
• The paper develops DataSpace-Builder, an execution-grounded framework that transforms executable Text-to-SQL resources into task-local heterogeneous workspaces via cross-language transformation, constraint-aware relational sampling, modality routing, artifact rendering, and human expert review and repair.
• The paper presents a deterministic evaluator that performs header-invariant column alignment, type- and precision-aware normalization, and order-aware row comparison, enabling exact verification of complete tabular results while accepting equivalent representations and rejecting incomplete or erroneous outputs.

Introduction

Data agents promise to serve as natural-language interfaces for organizational data, but in realistic analytical scenarios a user’s question often spans multiple data sources, languages, and formats. Existing benchmarks each address part of this challenge: structured-data tasks typically assume the right table or database is already identified, unstructured-data benchmarks focus on retrieval and factoid answers, and data-agent benchmarks lack a unified framework that combines a task-local heterogeneous workspace, a consistent output contract requiring a complete tabular result, and deterministic evaluation capable of accepting equivalent representations. The authors introduce DataSpace, a benchmark that forces an agent to autonomously discover and combine evidence from a workspace of CSV, JSON, SQLite, Markdown, PDF, and video files across Chinese and English, and to return a full typed table that is scored against a reference answer. They also contribute DataSpace-Builder, an execution-grounded pipeline that transforms relational Text-to-SQL resources into cross-language multimodal tasks, and a semantics-aware evaluator that tolerates column reordering and value normalization while rejecting incomplete or erroneous outputs.

Dataset

DataSpace is a benchmark of 410 heterogeneous workspace tasks that evaluate an agent’s ability to understand, align, compute over, and materialize results from mixed-modality, cross-language analytical environments. The authors construct it from English Text-to-SQL corpora and then transform, sample, render, and manually review each instance.

• Dataset composition and sources

  • Total tasks: 410
  • Source corpora: 363 tasks (88.5%) from BULL (financial) and 47 (11.5%) from EHRSQL (clinical)
  • Domains: fund (158), stock (120), macroeconomic (85), healthcare (47)
  • Language settings: 265 cross-language (question and database languages differ), 145 single-language
  • Workspace modalities: tasks combine CSV, JSON, SQLite, Markdown, PDF, and video in 13 modality combinations; every task includes CSV, and JSON, SQLite, Markdown, PDF each appear in over 93% of tasks; video appears in 189 tasks
  • Scale: 25,384 PDF pages, 55.36 million PDF/Markdown characters, 5.49 hours of video; reference answers total 126,409 rows (max 12,962 rows per task, max 6 columns), 92 tasks require ordered results

• Key details for each subset

  • Fund analytics: 158 tasks, BULL-sourced
  • Stock analytics: 120 tasks, BULL-sourced
  • Macroeconomic: 85 tasks, BULL-sourced
  • Healthcare: 47 tasks, EHRSQL-sourced
  • Cross-language subset: 265 tasks, processed by joint translation of database, question, and SQL
  • Single-language subset: 145 tasks
  • Filtering: all tasks pass automatic execution and semantic checks, then blind dual-expert human review

• How the paper uses the data

  • DataSpace is used as a fixed evaluation benchmark, not a training set
  • Each task couples four capabilities: workspace discovery, interpretation and alignment of types/schemas/entities/units/languages, relational computation (filtering, joining, aggregation, ordering, temporal reasoning), and complete tabular-result materialization
  • Evaluation uses human-verified gold answers and canonical configurations that specify column types, numeric precision, and ordering

• Dataset construction pipeline

  • Cross-Language Transformation: jointly translates the question, database, and SQL from English into Chinese/English combinations; linked columns (by foreign key, name, or value overlap) are translated consistently, with identifiers/dates/numbers frozen; a replacement map is stored; execution equivalence and LLM-based semantic alignment checks reject invalid instances
  • Constraint-Aware Relational Sampling: samples a task-local database from the transformed instance while preserving table inventory and schema; a safeguard set retains primary/foreign keys, join columns, predicate values, and target entities; sampled rows are propagated across relationships; the SQL is re-executed on the sampled database; samples are accepted only when execution succeeds, relationships remain valid, and the result is not degenerately empty
  • Modality Routing & Artifact Rendering: converts sampled tables into heterogeneous workspace artefacts (documents, video); details not provided in the excerpt, but the aim is to create varied modality combinations
  • Human Review & Task Repair: two independent domain experts solve the task using only the question and workspace, then verify the gold answer and author an evaluation configuration; agreement is required; disagreements trigger evidence-based repair and rechecking until consensus; unresolved tasks are removed

Method

The authors introduce DataSpace-Builder, a four-stage pipeline designed to transform existing Text-to-SQL instances into reviewed, heterogeneous workspace tasks. As shown in the figure below, the framework systematically processes source corpora through Cross-Language Transformation, Constraint-Aware Relational Sampling, Modality Routing & Artifact Rendering, and Human Review & Task Repair to produce finalized benchmark records.

In the first stage, Cross-Language Transformation, the authors perform a joint migration of the question, database state, and executable workload to handle cross-language scenarios. To maintain consistency across entities, they link columns connected by foreign keys or shared names and translate them jointly, generating table, column, and cell-level mappings $M = \{ M_{tab}, M_{col}, M_{val} \}$M={Mtab​,Mcol​,Mval​}. The transformed database and SQL are deterministically rewritten using these mappings. The pipeline retains a transformed tuple only after verifying structural validity, execution equivalence, and semantic alignment between the question and the SQL.

Next, Constraint-Aware Relational Sampling diversifies the task-local data to prevent repetitive entities. The authors construct a safeguard set $C_s$Cs​ combining primary keys, foreign keys, and predicate values extracted from the SQL abstract syntax tree. A relationally consistent sampling policy retains rows required by these safeguards and propagates key values across schema relationships. The sampled tables are materialized into an intermediate database $\mathcal{D}_s$Ds​, and the query is re-executed to generate a candidate reference result.

The Modality Routing & Artifact Rendering stage determines how agents encounter the relational content. A rule-based policy assigns sampled tables to compatible renderers, producing structured artifacts like CSV, JSON, and SQLite. For document artifacts, the system generates fact-grounded Markdown and PDF documents by planning document styles and assembling generation blocks. Additionally, for query-conditioned video rendering, the renderer derives typed evidence atoms from the executable SQL and sampled results, constructs a storyboard, and integrates the rendered video into the task workspace.

To ensure task quality, the final stage involves Human Review & Task Repair. Two domain experts independently solve the candidate task using only the question and workspace, then verify the gold answer and author the evaluation configuration $c_i$ci​. Any disagreement triggers an evidence-based discussion and minimal repair, followed by rechecking until consensus is reached.

The resulting benchmark establishes a rigorous task-level input-output interface where a data agent must autonomously discover and combine data across various modalities. Refer to the figure below illustrating the agent workflow as it interacts with the heterogeneous workspace, executes multi-step computations, and returns a complete tabular result.

Experiment

The study evaluated data agents on the DataSpace benchmark through two controlled comparisons: six multimodal backbones were tested with a fixed agent, and five agent harnesses were compared while holding the backbone constant. The benchmark proved unsaturated, with the best backbone solving only about two-thirds of tasks, and performance consistently degraded on tasks requiring cross-modality integration or joins. Harness design also significantly affected accuracy, and failure analysis showed that most errors arose from incorrect answer materialization and misunderstanding of the required output schema rather than from evidence retrieval or computation.

The DataSpace task interface requires agents to combine evidence from video, PDF, and SQLite, align entities, and compute metrics to produce a tabular answer. Experimental results show that cross-modality integration and join operations are the most consistent sources of accuracy degradation, with multimodal tasks underperforming single-modal ones for every backbone. Failure analysis reveals that the majority of errors stem from materializing the correct output schema rather than from evidence retrieval, with column misprojection and intent misformulation accounting for over half of audited failures. Required video evidence helps GPT and Kimi but hurts MiMo, Claude, and MiniMax, showing that cross-modality integration is a consistent challenge across models. Join requirements reduce accuracy for all backbones by 9.7–19.8 points, making them a uniform source of degradation. Answer materialization errors, such as adding or omitting columns, account for 52.2% of failures from the strongest model. Target-result misunderstanding and faulty column projection account for 56.6% of audited failures, outnumbering evidence selection errors. Multimodal tasks underperform single-modal tasks for every backbone by 1.8–14.0 points, highlighting the difficulty of integrating disparate evidence sources.

A survey of representative benchmarks reveals that no existing benchmark simultaneously provides cross-artifact integration, autonomous workspace discovery, long-document processing, cross-language inputs, and verifiable evaluation with complete table outputs and schema invariance. The DataSpace benchmark fills these gaps, enabling controlled study of agent failures across heterogeneous workspaces. Structured-data benchmarks such as Spider and BIRD offer database querying with exact answers but lack cross-artifact, discovery, and long-document requirements. Unstructured benchmarks like HotpotQA and CRAG include document retrieval and multi-hop reasoning, yet do not require complete table outputs or schema-invariant evaluation. Data-agent benchmarks such as KramaBench and FDABench introduce workspace discovery and multi-modality but still omit cross-language input, complete table verification, or schema-invariant scoring. Cross-modality integration and joins are the most consistent sources of accuracy degradation: multimodal tasks underperform single-modal tasks by up to 14.0 points, and joins reduce accuracy by 9.7–19.8 points across all evaluated backbones. Answer materialization errors, particularly faulty column projection, dominate agent failures, accounting for over half of audited mistakes and underscoring the challenge of respecting exact output schemas even when the underlying values are correctly computed.

DataSpace comprises 410 tasks with a median of 20 artifacts per task, totaling 15.01 GB of storage. The workspace includes 1,088 PDFs (25,384 pages), 875 Markdown files, and 189 videos lasting 5.49 hours, with high variability in per-task scale shown by P90 values. The median task has 20 artifacts, but the top 10% of tasks have at least 23 artifacts, with a range of 5 to 26. PDFs are the largest textual modality: 1,088 PDFs generate 25,384 pages and 28.48 million characters, with a median of 22 pages per PDF. Video content is concise: 189 videos average 1.7 minutes each, with a median duration of 103.8 seconds and a range of 39.1 to 158.3 seconds. Storage per task is skewed: the median is 31.26 MB, while the P90 reaches 74.18 MB, reflecting the diverse scale of workspaces.

When the agent harness is fixed, Grok 4.5 achieves the highest accuracy at 66.34%, yet 76 tasks remain unsolved by all six backbones, showing the benchmark is far from saturated. Keeping the backbone constant reveals that harness choice alone shifts accuracy by 15.36 points, demonstrating that both the model and the tooling substantially affect task completion. The top two backbones, Grok 4.5 and GPT-5.6 Sol, differ by only seven correct tasks (66.34% vs. 64.63%). Accuracy across the six backbones spans a 37.80-point range, from 28.54% to 66.34%. 76 tasks are missed by every backbone, while the oracle union of all backbones solves 81.46% of the benchmark. With the same backbone (MiMo-V2.5), harness accuracy ranges from 30.98% to 46.34%, a 15.36-point gap. Join requirements consistently reduce accuracy by 9.7–19.8 points across all backbones, and cross-modality integration degrades performance by 1.8–14.0 points for every model.

Answer materialization is the largest source of Grok 4.5 errors, followed by intent misformulation, and together these two incorrect output schema issues cause the majority of failures. The same evaluator symptoms often arise from different upstream causes, as column mismatches are mostly materialization errors but some originate in earlier stages, while no-submission results are rarely pure execution failures. These findings highlight that faithfully producing the requested output structure is a greater challenge than locating or computing the correct data. Materialization errors account for over half of all failures, primarily by adding or omitting columns after the correct answer is internally available. Intent misformulation contributes nearly one-quarter of failures, frequently misinterpreting the required output format or row granularity. Column-count mismatches are dominated by materialization issues (58 of 74), yet a notable minority stem from earlier intent, extraction, grounding, or computation errors. Only 5 of 13 no-submission outcomes are pure execution-control failures; the majority follow upstream errors in interpretation, extraction, grounding, or computation.

The DataSpace benchmark evaluates agents on tasks that require integrating evidence from video, PDF, and SQLite sources and producing a structured tabular answer, with experiments examining how cross-modality reasoning and join operations affect accuracy. The key findings show that answer materialization, particularly misprojecting columns or misinterpreting the output schema, is the dominant failure mode rather than evidence retrieval, and that both the underlying model and the agent harness independently contribute to large performance variations. Overall, the benchmark reveals that faithfully adhering to exact output formats while fusing heterogeneous data remains a persistent challenge for current agents.