One-sentence Summary

The Astribot Team presents Lumo-2, a latent world-action model that employs a multi-stage modality pre-alignment strategy to align action representations with latent world dynamics, vision, and language, overcoming the misalignment of reconstruction-based tokenization and enabling predictive reasoning and scalable control for challenging real-world tasks that demand temporal reasoning and physical understanding.

Key Contributions

• Lumo-2 is a latent world-action model that generates actions by reasoning over learned world dynamics in a compact latent space, enabling future-aware decision-making without explicit pixel-level rollouts at inference.
• A multi-stage modality pre-alignment strategy progressively aligns action representations with latent world dynamics, vision, and language, enforcing cross-modal consistency and inducing a semantically structured latent space that improves scaling properties.
• Systematic experiments on real-world manipulation tasks show that Lumo-2 achieves consistent gains over vision-language-action and world-action model baselines, with pronounced improvements on long-horizon and dexterous tasks, and analyses reveal better scaling behavior and out-of-distribution generalization.

Introduction

The authors address the challenge of building generalist robot policies that can actively project future physical possibilities rather than just memorize training data. Prior vision–language–action (VLA) models like Lumo-1 used explicit textual reasoning chains, which suffered from limited flexibility, poor scalability, and high inference latency. More broadly, aligning heterogeneous modalities—continuous actions, high-dimensional visual observations, abstract language, and latent world dynamics—is difficult because naive joint training leads to instability and poor generalization. The authors introduce Lumo-2, a latent world–action model that replaces explicit reasoning with a compact, physically grounded latent dynamics space. A three-stage progressive training paradigm aligns action tokens with latent world dynamics, vision, and language, treating the action modality as a first-class citizen. This enables future-aware reasoning without costly pixel-level rollouts and improves scaling behavior for end-to-end robotic control.

Dataset

The authors construct a heterogeneous, multi-source dataset to train the Lumo-2 model, spanning robot manipulation, egocentric human videos, general internet videos, and a large-scale vision-language corpus.

Robot manipulation data

• Self-collected Astribot S1 data with action speeds comparable to human demonstrations.
• Cross-embodiment datasets from diverse platforms: bimanual humanoids (AGIBot Genie-1, Galaxea R1 Pro, RealSource RS-02) and tabletop dual-arm robots (ARX, Agile X, YAM).
• A distribution debiasing method is applied to cross-embodiment data to mitigate action-rate ambiguity.

Egocentric human video data

• Sourced from EgoTaskQA, EPIC-KITCHENS, and Ego4D.
• In Stage 1 training, these videos are processed only through the vision branch, providing rich world dynamics without action labels.

General internet videos

• Used in Stage 3 for learning world dynamics from natural scenes.

Vision-language data (VLM corpus)

• General multi-modal understanding: the FineVision dataset (24.3M image-text samples) serves as the primary instruction-tuning set, supplemented with a large-scale video understanding corpus for spatiotemporal dynamics.
• Localization: object, area, affordance localization, and trajectory prediction with standardized coordinate outputs.
• Planning: robotic manipulation sequences from prior Lumo-1 data and the EO-Data1.5M dataset.
• Cognition: object attribute descriptions, spatial reasoning (size, distance, viewpoint), counting, and egocentric task understanding, built from ScanNet, ScanNet++, and Molmo-2.

Data processing and usage

• All vision inputs are 5-frame temporal sequences, resized to 224×224, sampled every 8 frames from 30 FPS video.
• Robot data uses multi-view fusion (head view queries wrist views); egocentric human videos use only the head view.
• Action data is decomposed into 8 semantic groups (torso, bimanual arms, each split into translational, rotational, and gripper components) and temporally compressed from 32 to 4 frames to match the visual branch resolution.
• In Stage 1, a hybrid batching strategy is used: human egocentric data flows through the vision branch only, while robot data is jointly trained on both vision and action branches.
• In Stage 3, VLM data, video data, and robot data are co-trained to integrate world knowledge, dynamics, and robot behaviors.

Method

Lumo-2 is an end-to-end, historical context-aware latent world-action model that jointly models distributions over latent world dynamics, action chunks, and textual outputs. Rather than directly predicting actions from observations, the model first infers latent world dynamics $\phi$ϕ and then conditions action generation on this latent representation. Both the high-level distribution over latent world dynamics and the low-level action distribution are parameterized within a single, unified multi-modal transformer.

Modeling action generation from a single-timestep observation is inherently ill-posed due to partial observability and ambiguity in action inference. For instance, in a typical "pour water" task, identical instantaneous observations may correspond to different underlying phases of execution. Panels showing the environment before and after pouring transparent water share similar visual observations, and a single observation during an ongoing pour does not indicate whether the action should continue.

To resolve this action ambiguity and capture temporally coherent control priors, the authors incorporate a short-term memory mechanism. They augment the observation with a temporal context buffer $\omega_{t-H':t}$ωt−H′:t​, which provides a compact summary of recent history, enabling the model to better infer latent dynamics. The imitation training objective is extended to condition on this context buffer alongside the observation and instruction.

To address the misalignment between reconstruction accuracy and downstream control performance in action tokenization, the authors propose a three-stage progressive training paradigm.

Stage 1: Aligning Actions with Latent World Dynamics The objective is to construct a physically grounded latent world dynamics space and obtain an action encoder with global context awareness. The authors adopt a co-training paradigm with feature fusion and bidirectional constraints to enable early alignment between world dynamics latent $\phi$ϕ and action latent $A$A. They enforce visual representations to contribute to action reconstruction, preserving visual components predictive of physical motion while suppressing nuisance signals.

Stage 2: Aligning Actions with Vision and Language Building on the dynamics-action alignment from Stage 1, Stage 2 explicitly incorporates visual and language modalities to transform learned action tokens into semantically enriched representations.

The authors augment the Stage 1-trained action encoder with a semantic module, transforming the original action latent into a semantically enriched latent represented by a new codebook. This module leverages visual dynamics as a prior constraint, endowing action tokens with semantic interpretability. To achieve cross-modal alignment, they employ multi-task learning with objectives including action reconstruction, behaviour comprehension, vision-language guided action generation, cross-modal prediction, and cross-modal action contrast and discrimination.

Stage 3: Co-Training on VLM, Video, and Robot Data To develop a generalist robotic foundation model that preserves rich world knowledge, the authors construct a large-scale dataset of high-quality vision-language data. The dataset places particular emphasis on localization, cognition, and planning, while also incorporating diverse data for general-purpose multi-modal understanding.

The curated dataset is designed to enhance core embodied reasoning abilities while preserving the general multi-modal understanding and reasoning capabilities of the pre-trained VLM. It includes data for general multi-modal understanding, localization covering object, area, affordance, and trajectory prediction, manipulation planning, and cognition encompassing object understanding, spatial reasoning, counting, and egocentric task understanding.

Finally, to improve inference speed for real-time robot control, the authors adopt Block-wise Autoregression (BAR). Instead of standard next-token prediction, BAR predicts the next block of tokens in a single forward pass by replacing the standard causal mask with a block-wise causal mask that permits intra-block attention, significantly reducing end-to-end inference latency while preserving autoregressive consistency.

Experiment

The experiments validate Lumo-2 across five dimensions: embodied reasoning evaluated via VLM benchmarks and co-training, latent world dynamics utility through probing and generalization tasks, few-shot post-training on diverse manipulation challenges, emergent human-to-robot transfer from heterogeneous data, and modality alignment with scaling analysis. The multi-stage training yields consistent improvements in spatial perception, future state prediction, and temporal reasoning, with latent dynamics capturing semantically meaningful traces that enhance out-of-distribution generalization. On challenging manipulation suites, Lumo-2 outperforms strong baselines in physical understanding, long-horizon execution, and dexterous control, while co-finetuning with human demonstrations provides clear gains on unseen objects and modality pre-alignment improves action reconstruction quality and downstream scaling.

The multimodal data corpus for Lumo-2 is organized into four categories: General Understanding, Localization, Planning, and Cognition. It draws on publicly available datasets, some of which are filtered or cleaned, and emphasizes localization, cognition, and planning while still covering general multimodal understanding. Compared to Lumo-1, the dataset substantially expands video data for spatial reasoning and egocentric understanding, and enriches the FineVision corpus to preserve the base model's general vision-language capabilities. The dataset places particular emphasis on localization, cognition, and planning, alongside general-purpose multimodal understanding. Lumo-2 substantially expands video data with a focus on general multimodal understanding, spatial reasoning, egocentric understanding, and localization, and uses a richer FineVision corpus to preserve general vision-language skills.

Lumo-2's vision-language model (VLM) achieves consistent gains over the Qwen-3.5 4B backbone on embodied understanding benchmarks, with particularly large improvements on tasks like VSIBench and MindCube. Co-training with action data (Stage 3) preserves these visual-linguistic skills and even lifts performance on BLINK-Mean, CV-Bench, and Ego-Plan2 compared to the VLM-only variant, indicating no catastrophic forgetting. The model also remains competitive with other 4B-scale embodied systems. Lumo-2 VLM raises VSIBench accuracy from 21.73 (Qwen-3.5 4B) to 52.34, highlighting the effectiveness of the curated embodied multimodal dataset. After multi-modal co-training, Stage 3 outperforms the VLM-only version on BLINK-Mean, CV-Bench, and Ego-Plan2, showing that integrating action signals does not degrade vision-language abilities.

VLWA fine-tuning consistently outperforms VLA fine-tuning on temporal reasoning tasks. It achieves perfect success on the conveyor belt task and a notable advantage on the rotating rack task, indicating that the VLWA paradigm yields stronger temporal reasoning capabilities. VLWA fine-tuning reaches 100% success on Collect Eggs from the Conveyor Belt, surpassing VLA fine-tuning by 8 percentage points. On Place Cubes on the Rotating Rack, VLWA fine-tuning achieves 81.67% success, outperforming VLA fine-tuning (74.17%) by a clear margin.

Evaluation tasks are organized into three capability dimensions: temporal reasoning, physical understanding, and control complexity. Temporal reasoning includes reactive, predictive, and memory sub-categories, each demanding specific temporal processing skills. Control complexity spans long-horizon execution and dexterous manipulation, emphasizing robustness and precision. Temporal reasoning and physical understanding tasks are trained with the full VLWA paradigm, while static manipulation tasks use action supervision only. Long-horizon tasks use two complementary temporal context schemes to mitigate perceptual aliasing: compact historical action tokens and visual trajectory traces.

The evaluation suite comprises real-world tasks designed to test dexterous manipulation, each broken into a sequence of meta-subtasks. Scoring is based on completing individual sub-steps, so the final task score reflects the degree of completion rather than a binary success or failure. All tasks demand fine motor skills such as picking, inserting, cutting, zipping, and handling deformable objects. Every task is classified under dexterous manipulation, requiring multi-step coordination of precise hand movements. One point is awarded per completed meta-subtask, providing a granular measure of partial task progress.

Lumo-2's multimodal dataset, which emphasizes localization, cognition, and planning with expanded video data, yields substantial gains over the Qwen-3.5 4B backbone on embodied understanding benchmarks. Co-training with action signals preserves visual-linguistic capabilities and even improves performance on several vision-language tasks, while the VLWA fine-tuning paradigm demonstrates stronger temporal reasoning than VLA fine-tuning on real-world tasks. The evaluation framework spans temporal reasoning, physical understanding, and dexterous manipulation, with long-horizon tasks using complementary temporal context schemes and real-world tasks scored by partial completion of fine-grained substeps.