One-sentence Summary

A Peking University and NVIDIA team presents PyRUA-DataFarm, an LLM coding agent that uses one calibration episode to rebuild tasks in simulation, with a video model rendering real-ready multi-view action-labelled data, yielding policies that match teleoperation-trained policies on five real-robot tasks (93 vs. 84 successes in 100 trials) and generalize better under unseen conditions (87 vs. 46 successes), while adapting a task costs $21 on average and each further episode costs$21onaverageandeachfurtherepisodecosts0.80.

Key Contributions

• The paper presents DataFarm, a system in which a coding agent converts a data request and one calibration episode into a simulation pipeline containing a scene, an expert controller, and a success judge, then uses a video model conditioned on real images to render synchronized, action-labeled multi-view robot data.
• The method introduces an independent admission process in which hidden judges, a qualified simulator, and independent reviewers certify the expert and every rendered episode before admission, so the generated data do not depend on trusting the agent.
• On five real-robot tasks, policies trained only on DataFarm data match teleoperation-trained policies with 93 versus 84 successes in 100 trials and retain 87 successes under unseen tablecloth and colored light where teleoperation-trained policies fall to 46. Adapting a task is fully automatic and costs $21 of API usage on average, plus$21ofAPIusageonaverage,plus5 for a render contract; every further episode costs $0.80 on average.

Introduction

Teleoperation provides faithful robot training data because every recorded action is executed by the robot and every image shows the deployment scene, but it is expensive and does not scale across new tasks, scenes, or viewpoints. Prior synthetic-data pipelines reduce collection cost but still require human engineering for each task, and they face a visual gap: simulation reconstructions struggle with deformable objects and lit displays, while video generation models lose exact action labels. The authors introduce DataFarm, an agentic data farm in which a coding agent compiles a task from a single calibration episode, a simulator supplies exact actions and states, and a real-image-conditioned video model supplies realistic multi-view appearance, with certification separated from production. This yields synchronized, action-labelled, real-ready robot data at low cost, and policies trained only on farm data match teleoperated-data policies in the original scene while generalizing better to unseen appearance.

Dataset

The authors describe DataFarm, a system that turns a data request and one calibration episode into a verified, action-labelled dataset.

Request and data modes

• A request is defined as $\rho = (e^c, \ell, n, \mu)$ρ=(ec,ℓ,n,μ), where:
  • $e^c$ec is a calibration episode: a short multi-view recording of the target scene.
  • $\ell$ℓ is a natural-language task description and success condition.
  • $n$n is the number of requested episodes.
  • $\mu$μ is the data mode.
• The calibration episode identifies the scene and cameras but is never used as a source of actions.
• Two data modes are supported:
  • sim2real: reproduces the recorded scene.
  • ood(V): additionally permits appearance variations, such as changing the tablecloth.
• No adapter, judge, or task-specific prompt exists before the request.

Episode schema

Each delivered episode $d_i$di​ contains:

• $o^i$oi: photographic multi-view observations.
• $a^i$ai: simulator actions.
• $x^i$xi: simulator states.
• $k^i$ki: contact events.
• $y^i$yi: task labels.
• $q^i$qi: admission evidence.

The dataset also includes a ledger $L$L that records every attempt and its outcome, including failed attempts.

Construction pipeline

The farm compiles each task through several intermediate representations:

1. Grounding: produces a scene description $G$G.
2. Simulator qualification and task compilation: produces a task contract $T$T and expert $C$C.
3. Rollout: executes an episode $E$E under a sampled variation $z$z.
4. Rendering: produces observations $O$O using a video model $p$p.
5. Verification: admits the final episode $d$d with evidence $q$q.

Each representation is produced by one stage and checked by a different component to avoid self-assessment by language models. If a defect is found, the rejection is sent back to the earliest responsible stage.

Task and expert rules

• The task contract lists task stages, completion conditions, and forbidden shortcuts, such as teleporting objects or relaxing the success criterion.
• A separate role turns the contract into a frozen judge before the expert is written.
• The expert is developed in a sandboxed harness shared across tasks.
• During development, the expert author may request rollouts on development seeds and see judge verdicts, but cannot read the judge or run held-out admission seeds.
• An admitted trajectory must be a single continuous execution, run afresh by the harness, that passes the judge.
• Each produced episode samples object and sensor variation, by default including small changes in table height and tilt.
• Joint occupancy is recorded so that the action labels remain executable on the real robot.

Filtering and admission

Automatic checks include verifying that the rendered video:

• Contains the correct objects and no extra objects.
• Shows events at the times indicated by the simulator.
• Is temporally stable and continuous across chunks.
• Has a static background.
• Matches the real scene appearance.
• Does not copy layout from a reference frame.

Verifier specifications are calibrated using deliberately corrupted versions of accepted episodes. Corruptions include deleted, duplicated, or enlarged objects, shifted events, swapped views, and changed colours. Each corruption must be caught by the responsible check.

After automatic checks, an independent reviewer compares the decoded video with real reference frames and the simulated state, then records evidence for each verdict.

Use in the model

Policy training starts only from a dataset whose admission record has been verified. Delivered datasets must be complete, unmodified, backed by the full attempt record, and reviewed episode by episode. The provided sections do not state explicit dataset sizes or training split ratios; dataset size is controlled by the request parameter $n$n.

Method

The authors introduce DataFarm, an agentic data farm that turns a data request and a single calibration episode into a stream of verified, action-labeled episodes. The system separates what is shared across tasks from what is produced for each task. The shared part is the capability layer $K$K, which comprises skills for the stages, role instructions, render templates that no task may override, and the harnesses, ledgers, and verifiers that execute and check the work. Every run is pinned to a fixed version of $K$K. For each request, the agent generates the task artifacts:

$$\mathcal{A}_{\rho} = \{G, T, S, C, J, R, B\}$$Aρ​={G,T,S,C,J,R,B}

namely a grounding record $G$G, a task contract $T$T, a sampler $S$S of scenes and initial states, an expert controller $C$C, a judge specification $J$J, a render contract $R$R, and the batch and dataset configurations $B$B. This separation ensures that a missing adapter is never a missing input, as the agent produces every artifact itself.

The farm compiles each task through a fixed sequence of intermediate representations, much as a compiler lowers a program. The grounding $G$G describes the scene, the task contract $T$T and the expert $C$C define how the task is solved, the episode $E$E records one execution under a sampled variation $z$z, the observation $O$O is its rendering by a video model $p$p, and the admission record $q$q certifies the delivered episode $d$d. Each representation is produced by one stage and accepted only after a component other than its producer has checked it, because language models are unreliable judges of their own output. A rejection is sent back to the earliest stage responsible for the defect, and every failed attempt is kept, so the final dataset comes with a complete record of how it was obtained.

The process begins with Grounding, which turns the calibration episode into a description of the scene that the simulator can reproduce. From the full episode, the agent identifies the robot and the objects, estimates their geometry and articulation, and fits the cameras and the metric scale. Physical quantities that a short video cannot determine, such as mass or friction, are not guessed; every quantity is labeled as measured, fitted, assumed, or unknown, and uncertain ones are covered by randomization in the simulator.

Before the agent writes an expert, it performs Simulator Qualification to check that the simulator can reproduce the skills the task requires. The farm runs short probes of these skills over candidate simulator variants that adjust material, contact, or actuator parameters, and selects the variant with the fewest changes that passes every probe.

Next is Task Contract and Expert Compilation. The agent specifies the task and writes an expert that solves it. The farm prevents the expert from fitting its own checks to avoid reward hacking. The task contract $T$T lists the stages of the task, the conditions that mark their completion, and the shortcuts that are forbidden. A separate role turns the contract into a judge, which is frozen before the expert is written. The expert is developed in a sandboxed harness shared by all tasks. The author writes a controller as a program and requests rollouts on development seeds, receiving the judge's verdict on each, but it can neither read the judge nor run the held-out seeds on which the expert is finally admitted.

Rendering then turns each simulated episode into photographic video while leaving its actions, objects, and timing unchanged. The data either reproduce the recorded scene or vary its appearance along axes the request authorizes. For each task, the agent writes a render contract that states the target appearance, the permitted variations, and real reference frames for every camera. The default renderer is MiniMax-H3 in its reference-to-video mode. Edge maps of the simulator video drive a ControlNet that fixes the arm pose, the object positions, and the timing, while real frames and a picture of the real robot supply the appearance. All cameras in one generation are tiled on one canvas and generated as a single video, so they share lighting and materials. Long episodes are generated as independent chunks and joined where neighboring chunks differ least to avoid drift.

Finally, Verification, Independent Review, and Admission ensure data quality. An episode enters the dataset only if an independent reviewer accepts it after automatic checks. The verifier is a single program shared by all tasks and configured by a task-specific specification. It checks that the rendered video contains the right objects, that events happen when the simulation says they do, and that the video is temporally stable. Before a specification is used, it is calibrated on deliberately corrupted versions of an accepted episode. Every episode is also examined by an independent reviewer, a separate agent that compares the decoded video with the real reference frames and the simulated state. A final delivery check confirms that the dataset is complete, unmodified, backed by the full record of attempts, and reviewed episode by episode. Policy training starts only from a dataset whose admission record has been verified.

Experiment

The evaluation uses five bimanual manipulation tasks on a dual-arm PiPER robot, where each task starts from one calibration episode to generate 100 training episodes and compares policies trained on real teleoperation, simulation with domain randomization, and DataFarm rendered data. Farm-trained policies match or exceed teleoperation in the original scene and transfer much better to unseen tablecloths and lighting, while the conventional simulation baseline transfers poorly, showing that photorealistic rendering of exact simulated trajectories is the key contributor. Cost results indicate that farm data become cheaper than teleoperated collection after a small number of episodes, and rendering checks confirm that all camera views stay faithful to the simulated motion and look realistic, unlike direct video model calls, while improving hard-to-simulate object appearances at low cost.

Most compared pipelines rely on curated inputs such as multiple demonstrations, task descriptions, reconstructed scenes, or teleoperated data, and their simulation tasks are usually built by people or language models. DataFarm instead starts from one calibration episode of a real scene, reuses none of its actions, and has an agent build the simulation task. It keeps actions and states in simulation while generating photographic multi-view observations conditioned on that episode's own images. DataFarm is the only compared pipeline that builds a simulation task from a single calibration episode and reuses none of its actions. Unlike methods that observe simulator renderings, point clouds, Gaussian splatting, or rendered scans, DataFarm uses the calibration episode's own images as appearance reference for photographic multi-view video.

The evaluation covers five bimanual manipulation tasks that target distinct challenges: nesting rigid cups, placing an apple on a live-display scale, beating a drum, using a drawer, and folding a towel. All tasks are evaluated on a dual-arm PiPER robot under both in-distribution original scenes and out-of-distribution scenes with an unseen tablecloth and changed colored lighting. Success conditions vary by task, ranging from complete nesting and stable placement to repeated drum strikes, closing a drawer with the block inside, and folding a towel twice. The task set spans different simulation needs: rigid and articulated tasks run in SAPIEN, while the deformable towel runs in Isaac Sim. Each task is tested in both original and out-of-distribution conditions, with out-of-distribution trials adding unseen tablecloths and altered room lighting.

Across the five tasks, bringing a task from calibration through admission and a render contract to 100 simulated episodes costs between roughly $23 and$23and57 at standard API pricing. The one-time grounding and qualification stage is the largest cost component for most tasks, with fold towel the most expensive and weigh apple the least expensive. After setup, recurring rollout and audit costs are considerably lower, averaging under one dollar per admitted episode. Fold towel has the highest total cost among the five tasks, while weigh apple has the lowest. The one-time grounding-to-admission stage dominates per-task cost, and the 100-episode rollout and audit stage stays below $14 for every task.

Automated farm data collection has low per-episode running costs, with fixed adaptation and rendering-contract costs that spread out as more episodes are produced. At 100 episodes the average cost per episode is slightly above one dollar and falls toward the per-episode running cost at 1,000 episodes. First-hour synchronized multi-view data costs more than subsequent hours, and the farm is generally cheaper than teleoperation-based collection under the stated assumptions. Across the five tasks, the mean per-episode running cost is below one dollar, with fold towel the most expensive and beat drum the least expensive. Spreading one-time and contract costs over 1,000 episodes reduces the average cost per episode close to the per-episode running cost, from slightly above one dollar at 100 episodes. One hour of synchronized three-view data per task costs more for the first hour than for each additional hour because fixed adaptation and contract costs are front-loaded. Estimated North American teleoperation labor is substantially more expensive per usable hour than the farm's data cost, while a full-service quote from China is only slightly above the farm's first-hour average. The farm's cost advantage depends on self-hosted rendering; a published hosted renderer price would make per-episode rendering substantially more expensive.

Across the three tasks, direct video-model calls struggle to follow simulated motion and structure, with higher lag and more frequent jumps. Rendering all cameras together with the full pipeline greatly improves motion and structure agreement, removes lag, and sharply reduces jumps. The full self-hosted renderer performs comparably to the hosted Seedance model while avoiding multi-view inconsistencies. Direct one-canvas and per-camera generation have much lower motion and structure agreement with the simulator than the full renderer, with higher lag and more frequent jumps. Rendering each camera independently degrades multi-view consistency, producing worse correlations and more jumps even for the full renderer on stack cups. Edge control alone improves motion tracking but leaves many jumps, which the full pipeline with references, prompts, and chunk assembly reduces to very few. The full renderer achieves zero lag, low jump counts, and motion and structure agreement comparable to the hosted Seedance renderer.

The experiments evaluate DataFarm, a pipeline that builds simulation tasks from a single real calibration episode without reusing actions, across five bimanual PiPER manipulation tasks in original and out-of-distribution scenes. They find that one-time grounding and admission dominate setup costs while recurring rollout and audit costs remain low, and automated collection is cheaper than teleoperation when rendering is self-hosted. Ablations show the full multi-view rendering pipeline is essential for strong motion and structure agreement, eliminating lag and reducing jumps while matching hosted video generation quality without multi-view inconsistencies.