One-sentence Summary

Researchers from Peking University and Zhongguancun Academy propose DataFlow-Harness, a grounded code-agent platform that bridges the NL2Pipeline gap by guiding an LLM agent to construct persistent, editable DAGs through typed incremental mutations, integrating DATAFLOW-SKILLS for procedural guidance, an MCP layer that exposes the live operator registry and pipeline state, and DATAFLOW-WEBUI to synchronize conversational authoring with a visual editor, achieving a $93.3\%$93.3% observed end-to-end pass rate on a 12-task data-engineering benchmark with a $72.5\%$72.5% reduction in measured monetary cost and $49.9\%$49.9% lower generation latency relative to Vanilla Claude Code, while staying within $0.9$0.9 percentage points of the Context-Aware Claude Code baseline and cutting cost by $42.8\%$42.8%.

Key Contributions

• The NL2Pipeline gap is formulated as the disconnect between natural-language workflow intent and persistent, editable, platform-native workflow artifacts.
• DATAFLOW-HARNESS combines procedural Skills, live Model Context Protocol grounding, typed incremental mutations, structural validation, and a synchronized conversational and visual editor to construct platform-native directed acyclic graphs.
• On a 12-task data-engineering benchmark, DATAFLOW-HARNESS achieves a 93.3% observed end-to-end pass rate that is within 0.9 percentage points of the Context-Aware Claude Code baseline, while measured monetary cost is 72.5% lower than Vanilla Claude Code and generation latency is 49.9% lower; a per-task ablation shows that Skills are most beneficial when construction depends on implicit procedural knowledge.

Introduction

The authors address a growing challenge in deploying large language models (LLMs) for industrial data‑processing workflows. While coding agents can translate natural‑language requirements into executable scripts, those scripts are typically disposable, hard to audit through graphical interfaces, and frequently hallucinate dependencies or rely on unavailable operators. This creates what the authors call the NL2Pipeline gap: the disconnect between a user’s natural‑language intent and the persistent, editable, platform‑native pipeline artifacts required for production governance. Prior code‑generation approaches focus on task accuracy but fail to ground construction in the live semantics of a data‑engineering platform. The authors contribute DATAFLOW-HARNESS, a platform that closes this gap by guiding an agent to build structured workflows rather than raw code. It combines procedural Skills that encode domain knowledge, live Model Context Protocol access to the operator registry and current state, and a synchronized conversational/visual interface. The result is a pipeline‑authoring system that matches script‑generation pass rates while reducing token usage, cost, and latency, and produces reusable, visually inspectable DAG‑based artifacts.

Method

The authors organize the workflow synthesis of DATAFLOW-HARNESS around four core components: DATAFLOW-WEBUI, the MCP Tools Layer, the Data Pipeline Backend, and DATAFLOW-SKILLS. The operational lifecycle centers on the Data Pipeline Backend, which serves as the authoritative source of truth across conversational, visual, and programmatic interfaces. Pipeline mutations are issued through the MCP Tools Layer, validated and committed to the backend, and then synchronized with DATAFLOW-WEBUI. In parallel, DATAFLOW-SKILLS provides procedural guidance that shapes agent reasoning without directly modifying pipeline state.

The Data Pipeline Backend represents a pipeline as $P = (D, O, E, S, R)$P=(D,O,E,S,R), where $D$D is the set of data sources and their URIs, $O$O is the set of configured operator instances, $E \subseteq O \times O$E⊆O×O contains directed data-dependency edges, $S$S records input and output field schemas, and $R$R contains runtime state such as model-serving endpoints. Rather than generating free-form code, agents interact with the backend through typed mutations, including adding or removing operators, updating parameters, and connecting edges. A mutation is committed only if the resulting graph is acyclic and adjacent operator schemas are compatible. These checks establish structural validity, though they do not by themselves guarantee semantic correctness, endpoint availability, or output quality.

DATAFLOW-WEBUI provides two synchronized modalities for workflow construction: a conversational interface for natural-language authoring and a visual DAG editor for direct workflow inspection and editing.

In the conversational interface, users describe workflow requirements in natural language. Before each agent turn, the current pipeline state and the DataFlow operator registry are injected into the context of the underlying language model via MCP. Guided by DATAFLOW-SKILLS, the model interprets user intent and determines the required workflow modifications, which are expressed as MCP tool calls and applied to the Data Pipeline Backend. The visual DAG editor renders the workflow as a directed acyclic graph. Users can inspect agent-proposed changes, adjust parameters, relink edges, or add and remove operators directly. Any manual edit is immediately committed to the Data Pipeline Backend, ensuring that subsequent agent interactions operate on the latest workflow state without requiring explicit re-synchronization.

Every pipeline change, whether proposed by the agent or made manually, passes through a Request-Validate-Commit protocol within the MCP Tools Layer. At the start of each synthesis turn, the agent fetches the latest pipeline state, incorporating any manual edits since the previous turn. Guided by DATAFLOW-SKILLS, the agent issues an MCP tool call expressing the intended change as a typed, structured mutation grounded in the DataFlow registry's live metadata. The system then verifies two properties: that the updated pipeline remains a directed acyclic graph, and that the output field schema of each operator is compatible with the input schema of every downstream operator. Changes that fail either check are rejected. Validated changes are written to the backend store, and a WebSocket notification broadcasts the updated state to connected clients to keep the authoring modalities synchronized.

DATAFLOW-SKILLS provides procedural guidance for workflow synthesis by injecting domain-specific knowledge into the reasoning context. While the MCP Tools Layer exposes operator metadata and workflow state, it does not encode recommended construction strategies or operator-composition best practices. To address this, DATAFLOW-SKILLS encodes two classes of knowledge. The first consists of procedural blueprints that define recommended workflow-construction sequences, including schema inference, operator selection, parameter configuration, and serving verification. The second consists of compositional constraints that capture operator compatibility rules, such as modality matching and field-flow conventions for nested structures. Together, DATAFLOW-SKILLS guides agent reasoning, while the MCP Tools Layer grounds execution against the live DataFlow environment.

Experiment

The experiments compare free-form code generation with structured DAG synthesis across industrial data-processing tasks, finding that directly constraining agents to platform operators without procedural guidance (MCP-only) sharply reduces task success, while DATAFLOW-HARNESS nearly closes this gap and yields significant cost and latency savings. Ablation shows that procedural skills help most on ambiguous, procedure-intensive tasks but offer little benefit when workflow routing is trivial or failures arise from model limitations. Downstream evaluations further indicate that grounded pipelines produce higher-quality training data, leading to improved model accuracy on challenging math and code benchmarks, demonstrating that systematic operator reuse and procedural guidance improve both pipeline reliability and the downstream utility of the generated data.

Enriching vanilla code generation with execution context improves end-to-end success rate, but both techniques output monolithic scripts that consume substantial resources. Directly synthesizing native DAG workflows using only operator specifications introduces a clear reasoning challenge, causing a large drop in pass rate. DATAFLOW-HARNESS bridges this gap by incorporating procedural guidance, achieving success close to that of context-aware script generation while drastically reducing monetary cost and generation latency through compact workflow representations. Adding execution context to vanilla code generation lifts the pass rate by a few percentage points but increases input token consumption. Switching from scripts to native DAG synthesis with only operator specs causes a sharp decline in end-to-end success, highlighting the NL2Pipeline gap. DATAFLOW-HARNESS recovers most of the lost success, reaching a pass rate within one percentage point of the strongest script baseline. Compared to vanilla script generation, DATAFLOW-HARNESS cuts monetary cost by over 70% and generation latency by nearly half. Even against the context-aware script baseline, DATAFLOW-HARNESS reduces cost by over 40% and latency by nearly one fifth while maintaining comparable success. Workflow representations are markedly more token-efficient: using only operator specs nearly halves input tokens, and adding procedural guidance further reduces total token usage by about a quarter.

In textbook-to-VQA extraction, DATAFLOW-HARNESS outperforms all baselines on both precision and coverage. The largest gains occur in coverage, where it recovers far more valid question-answer pairs than alternatives. This suggests that the framework constructs more complete workflows rather than merely filtering outputs conservatively. DATAFLOW-HARNESS attains 97.2% precision and 87.3% coverage, the highest values among the evaluated methods. Coverage shows the largest absolute improvement, with the proposed method recovering substantially more extractable QA pairs than any baseline. A context-aware chain-of-thought baseline strongly outperforms the vanilla approach, nearly doubling coverage and highlighting the value of document layout awareness.

Procedural guidance strongly boosts success on tasks that require implicit domain knowledge to assemble workflows, lifting aggregate pass rates from 18/30 to 29/30 runs. Trivially routable tasks see no gain because both approaches already achieve perfect success. Failures on the remaining tasks persist equally, pointing to bottlenecks outside workflow synthesis where guidance does not help. On procedural-knowledge-dependent tasks such as QA generation and text-to-QA chains, DataFlow-Harness improved aggregate success from 18 out of 30 to 29 out of 30 runs. Both methods achieved a perfect 10 out of 10 pass rate on each trivially routable task, including field rename and nested flatten. Multi-field scoring tasks exhibit identical failure rates for both methods, indicating that the remaining errors stem from downstream numerical constraints rather than workflow construction.

Data synthesized by the DataFlow-Harness pipeline leads to consistently higher average accuracy on the math suite compared to the Vanilla CC pipeline at matched training epochs. After one epoch, average accuracy improves from 49.9 to 51.6, and after two epochs from 54.5 to 55.7. The gains are largest on the hardest benchmarks, notably AIME24 and AIME25, indicating the grounded pipeline produces more challenging and higher-quality reasoning traces. On AIME24@32, one epoch of training on DataFlow-Harness data raises accuracy from 25.1 to 35.9, a relative jump of over 40%. AMC23 accuracy more than doubles with DataFlow-Harness data at one epoch (72.5 vs. 47.5), showing strong improvement on competition-style math.

The DATAFLOW-HARNESS pipeline produces comparable knowledge performance and mixed math results compared to the Vanilla CC baseline, but delivers consistent gains across all code benchmarks. These coding improvements drive a 2.3-point increase in the overall nine-benchmark average, suggesting the grounded critique-then-rewrite and judging stages yield more executable, better-structured code. MMLU scores are nearly identical (74.2 vs. 74.4), showing no degradation in factual knowledge. All four code benchmarks favor DATAFLOW-HARNESS, with the largest gap on MBPP (75.4 vs. 64.6). The overall benchmark average rises from 61.5 to 63.8, driven primarily by coding task gains. Math benchmarks are mixed: GSM8K declines (79.5 vs. 82.9) while MATH and Olympiad see modest gains. The grounded pipeline's critique-and-filter stages likely produce more reliable code outputs.

The evaluation compares DATAFLOW-HARNESS against script-based generation and vanilla pipelines across workflow synthesis, information extraction, math data generation, and multi-benchmark training. The framework recovers most of the success lost when moving from monolithic scripts to native DAG synthesis, while cutting monetary cost by over 70% and latency by nearly half. Procedural guidance proves essential for tasks requiring implicit domain knowledge, and the pipeline produces higher-quality math and code data, with notable gains on challenging reasoning benchmarks. Overall, DATAFLOW-HARNESS delivers competitive or superior performance with markedly lower resource consumption.