One-sentence Summary

Researchers from Huazhong University of Science and Technology and ArcheOne AI propose World In-Context Learning (WICL) and the RoboScript infrastructure, in which a frozen model rehearses each manipulation task in simulation from a single demonstration, stores the resulting feedback as Task Memory in context rather than in weights, and deploys that memory on new scenes to achieve 83.4% mean success across all 50 RoboTwin 2.0 tasks, exceeding the strongest trained action policy by over 20 points.

Key Contributions

• This work introduces World In-Context Learning (WICL), a paradigm in which a frozen model starts from a demonstration, rehearses the task in a world such as a simulator, and writes feedback into context rather than weights. RoboScript implements WICL as a modular infrastructure for agentic manipulation that stores the learned knowledge as a Task Memory and enforces deployment data isolation.
• The knowledge learned in context is semantic, encoding task steps and object roles rather than visual appearance, so it transfers across visual distributions, works with three different foundation models, and improves through reflection.
• On all 50 RoboTwin 2.0 tasks, RoboScript reaches 83.4% mean success across clean and randomized settings with one demonstration per task and no gradient updates, exceeding the strongest trained action policy by over 20 points. The same learned memories transfer to a physical dual-arm robot through cross-embodiment memory adaptation without re-recording demonstrations or repeating rehearsal.

Introduction

Robot manipulation has traditionally relied on trained action policies, such as vision-language-action models or world action models, which store skills in network weights. These policies perform well within their training distribution but degrade sharply under distribution shift, and every new task or domain requires new data and gradient updates. In-context learning avoids retraining, but prior in-context manipulation methods lack feedback from the world, so demonstrations from other scenes may not transfer and errors can accumulate without correction. The authors formalize World In-Context Learning (WICL), where a frozen model learns a manipulation task by rehearsing in a faithful world, distilling outcomes into reusable semantic context, and deploying without weight updates. They implement this in RoboScript, a modular agent-centric manipulation platform with strict data isolation, enabling single-demonstration learning, recursive self-improvement, and cross-embodiment transfer.

Method

The authors introduce RoboScript, an implementation of World In-Context Learning (WICL) for robot manipulation. WICL extends in-context learning by incorporating feedback from a world, alternating between rehearsal and revision to update a context $C$C while keeping the foundation model weights $\theta$θ frozen. The learned context, termed Task Memory $C^\star$C⋆, captures semantic knowledge about object roles, workflows, and relative geometry, allowing transfer across different environments.

Refer to the framework diagram:

The RoboScript pipeline consists of four main stages: skill building, task learning, deployment, and reflection.

First, the agent builds a reusable skill set to establish its action interface. It perceives the environment through RGB-D images and proprioception, using geometric-first perception to detect and track objects. The agent acts through a three-layer API comprising shared entry points, environment bindings, and environment-specific skills. By segmenting a demonstration into action primitives, the agent identifies gaps and generates new skills via a coding agent, validating them until coverage stabilizes. Every action returns a receipt detailing the commanded and achieved poses, grounding the feedback $r$r in concrete observations.

Next, the agent learns a Task Memory from a single expert demonstration through two stages: Imitation (S1) and Rehearsal (S2). In S1, a reader role initializes the memory $C_0 = \mathcal{T}_\theta(d)$C0​=Tθ​(d), and an acting role practices on the demonstration instance while a reviewing role revises the memory upon failure. In S2, the agent rehearses across new simulation seeds, updating the memory via $C_{k+1} = \mathcal{U}_\theta(C_k, \tau_k, r_k)$Ck+1​=Uθ​(Ck​,τk​,rk​). Revisions are merged only if they lead to success, and a memory linter ensures the Task Memory remains strictly semantic, rejecting absolute coordinates or simulator-specific facts. This strict data isolation guarantees that the acting agent relies solely on deployable data.

During deployment, the agent operates with the frozen Task Memory $C^\star$C⋆ on unseen instances without receiving world feedback, executing $a_t \sim \pi_\theta(\cdot \mid o_t, C^\star)$at​∼πθ​(⋅∣ot​,C⋆). Before manipulation, it performs scene anticipation to map the semantic blueprint to the current layout and verify reachability. If an action fails, the agent recovers by moving to a safe pose and continuing without resetting the environment, bounded only by the step limit.

Finally, the system enables recursive self-improvement through a reflection loop. After evaluating on a disjoint block of instances, a reflection agent analyzes the episode records $D_j$Dj​, which contain only deployable observations and success indicators. It diagnoses failure patterns and generates a guidance document $G_{j+1} = \mathcal{V}_\theta(G_j, D_j)$Gj+1​=Vθ​(Gj​,Dj​) for the next block. This process allows the agent to improve its generalization capabilities over multiple rounds without gradient updates or privileged data.

Experiment

The experiments evaluate RoboScript on 50 dual-arm RoboTwin 2.0 tasks in clean and randomized settings, where the agent learns each task from one clean demonstration plus rehearsal on clean scenes and is tested in both conditions. Main results show that a single demonstration with rehearsal matches trained policies in clean scenes and substantially outperforms them under domain randomization, with gains attributed to semantic knowledge in context, demonstration anchoring, world interaction, rehearsal, and deployment recovery rather than privileged supervision. Additional experiments demonstrate recursive self-improvement from the agent’s own evaluations, report learning and inference costs, and show that the learned task memory transfers to a physical dual-arm robot with a different embodiment through parameter remapping.

RoboScript with a foundation-model agent and one demonstration per task outperforms trained action policies that use 50 clean demonstrations per task. Its clean and randomized success rates remain nearly equal, while trained policies lose most of their success under randomization. The advantage holds across different foundation models and stems from context-based memory rather than gradient updates or randomized training data. A single demonstration with rehearsal and no gradient update yields higher overall success than trained policies using 50 demonstrations per task. Foundation-model agents retain 95.2% to 100.8% of clean success under randomization, whereas trained policies retain at most 65.1%. The strongest trained policy is comparable to foundation-model agents in clean settings but drops sharply under randomization. Performance scales with model capability among foundation models, with the strongest model leading the others by about five points.

This ablation set isolates the roles of world interaction, demonstrations, and deployment components. The world and the demonstration are both necessary: removing either causes large drops, and rehearsal without a demonstration can be worse than no rehearsal. Deployment-time recovery is also important, and rehearsal adds the most on harder tasks with greater scene variation. World interaction after a demonstration produces a large gain over building a memory from the demonstration alone, while demonstration-only and instruction-only memories without world interaction differ by less than 1.5 points. Removing the demonstration before rehearsal yields the largest drop in the study, and rehearsal from the instruction alone underperforms deploying the same instruction-only memory without rehearsal by 11.5 points. Deployment-time recovery is the most important removed deployment component; removing it lowers success more than removing the arm or completion check. On harder tasks excluded from this set, rehearsal provides a larger average gain and an even larger gain under randomization, where scenes differ most from the demonstration.

Across the three foundation models, GPT-6.1-Sol achieves the highest average success rate but also the highest total cost, while GPT-5.6-Luna is the cheapest and DeepSeek-V4.1-Flash is intermediate. GPT-5.6-Luna passes the demonstration exit rule on the fewest tasks, whereas DeepSeek-V4.1-Flash passes on all tasks. Evaluation costs dominate total spending, and frozen Task Memory can be reused without additional learning cost. GPT-6.1-Sol has the highest average success rate but is the most expensive, while GPT-5.6-Luna is the least expensive. DeepSeek-V4.1-Flash passes the demonstration exit rule on all tasks but generates far more output tokens per call than GPT-6.1-Sol.

These experiments evaluate RoboScript, a foundation-model agent given a single demonstration and rehearsal, against trained action policies with many clean demonstrations, and isolate the contributions of world interaction, demonstrations, and deployment components. RoboScript retains nearly all of its clean success under randomized scenes whereas trained policies drop substantially, with gains coming from context-based memory rather than gradient updates and scaling with foundation-model capability. Ablations show the demonstration and world interaction are both necessary, deployment-time recovery and rehearsal contribute most on harder varied tasks, and cost results indicate the strongest model attains the highest success but is most expensive while frozen Task Memory can be reused.