One-sentence Summary

SimWAM, a simple world-action model from Huazhong University of Science & Technology and Dongfeng, co-trains a video expert and an action expert with flow matching and an isolated attention mask, using video generation purely as a training signal so that discarding the video branch after training leaves a self-contained planner that achieves 91.5 PDMS on NAVSIM, surpasses state-of-the-art WAM planners with substantially lower latency, and transfers zero-shot to nuScenes.

Key Contributions

• A training-time paradigm that uses joint flow matching to transfer motion priors from a pretrained video expert to a lightweight action expert, allowing the video branch to be discarded at inference and leaving a self-contained planner for direct trajectory prediction.
• A modular architecture with isolated attention masking and no shared parameters, enabling independent scaling and replacement of the video backbone without modifying the learning objective or inference pipeline.
• A reinforcement learning refinement stage with a compositional driving reward that optimizes the standalone planner beyond imitation, yielding 91.5 PDMS on NAVSIM, lower latency than prior WAM-based planners, and zero-shot transfer to nuScenes.

Introduction

End-to-end autonomous driving replaces classical modular pipelines with a single network that maps sensor observations directly to planned trajectories, reducing error propagation but relying heavily on imitation learning that captures traffic semantics, user intent, and scene dynamics only implicitly. Vision-Language-Action models improve semantic understanding by integrating pretrained knowledge, while world-action models incorporate video generation to explicitly model environment evolution. However, existing driving world-action models follow an imagine-then-act paradigm that couples costly future frame generation into the real-time planning loop, significantly increasing inference latency. The authors introduce SimWAM, a streamlined world-action model that co-trains a pretrained video expert and a lightweight action expert with flow matching, using an isolated attention mask to prevent the action expert from accessing future frames. After training, the entire video branch is removed, transferring traffic dynamics priors into a direct trajectory predictor that incurs no test-time generation overhead and can be further refined with reinforcement learning for driving-quality optimization.

Method

The authors present SimWAM, a world-action model for end-to-end autonomous driving. The overall framework is illustrated below.

SimWAM adopts a direct policy interface for trajectory planning. Given a front-camera observation $o_t$ot​, ego state $s_t$st​, and navigation command $l$l, the planner predicts an ego trajectory $a_{t+1:t+H}$at+1:t+H​ in the ego-vehicle coordinate frame. Unlike existing driving world-action models that rely on an imagine-then-act factorization, SimWAM expresses the policy as $p_\theta(a_{t+1:t+H} \mid o_t, s_t, l) = p_\theta(a_{t+1:t+H} \mid z(o_t), s_t, l)$pθ​(at+1:t+H​∣ot​,st​,l)=pθ​(at+1:t+H​∣z(ot​),st​,l), where $z(o_t)$z(ot​) is the representation from the current observation. This design ensures that inference requires neither future-scene latents nor auxiliary motion modules.

The architecture comprises two main components: a video expert and a lightweight action expert. The video expert is a video Diffusion Transformer initialized from a pretrained model, equipped with a video VAE and a T5 text encoder. The VAE maps driving frames into latent tokens, while the navigation command is integrated via T5 cross-attention. The current frame serves as a clean condition, and future frames are noised and reconstructed using flow matching. The action expert is a lightweight Diffusion Transformer conditioned on $c = \{z(o_t), s_t, l\}$c={z(ot​),st​,l}. It predicts the trajectory velocity field $v_{\theta_a}(a_{t+1:t+H}^\tau, \tau, c)$vθa​​(at+1:t+Hτ​,τ,c) via flow matching, where a small MLP embeds the ego state.

To exploit video generation during training while eliminating its computational cost at inference, the authors introduce an isolated attention mask. In the shared attention stream, the current observation latents, future frame latents, and action tokens are processed together. Both future frame tokens and action tokens attend to the current observation latents but remain mutually invisible. This structural modification decouples the action expert from the video branch, allowing the video Diffusion Transformer and future-frame decoder to be discarded after training. Consequently, the action expert remains self-contained and directly predicts trajectories from current inputs.

During the co-training stage, the two experts interact only through the shared attention stream. Joint flow matching over video and trajectory modalities allows future-scene prediction to shape the observation representation used for planning. The joint objective is defined as:

where $\mathcal{L}_{\mathrm{FM}}^{\mathrm{act}}$LFMact​ and $\mathcal{L}_{\mathrm{FM}}^{\mathrm{vid}}$LFMvid​ instantiate the flow matching loss on the action trajectory and future-frame latents, respectively, and $\lambda$λ balances the two terms. The base flow matching loss is given by:

To further optimize trajectory generation toward driving quality, the authors introduce reinforcement learning. Since the deterministic flow ODE lacks the stochasticity required for diverse maneuver exploration and tractable transition likelihoods, it is transformed into a marginal-preserving SDE:

where $\mathrm{d}w$dw is a Wiener increment. During the RL stage, a group of candidate trajectories is sampled for each scenario and evaluated using a compositional reward function. Group-relative advantages are derived for the clipped policy update, and only the LoRA adapters of the action expert are updated to preserve the distilled motion prior.

Experiment

SimWAM is evaluated on the NAVSIM planning benchmark using only a single front camera. Its two-stage training – joint learning with a video generation model followed by reinforcement learning on challenging scenes – yields state-of-the-art end-to-end planning performance without auxiliary motion modules. The isolated attention design decouples action prediction from future video, preserving efficient inference, while independent scaling of the action and video experts further strengthens results. Qualitative and cross-dataset evaluations confirm more decisive yet safe maneuvers and strong generalization with very low collision rates.

With only a single front camera, SimWAM achieves 91.5 PDMS on the NAVSIM navtest benchmark, setting a new state of the art for learned end-to-end planning. It surpasses the strongest VLM-based planner by 0.4 points and the image-prediction-augmented ExploreVLA by 1.1 points, while attaining the best driving area compliance and ego progress among world-model-based methods with competitive no-collision and time-to-collision scores. Using a single front camera, SimWAM reaches 91.5 PDMS, outperforming all prior learned planners including VLM-based and world-model-based approaches. SimWAM exceeds the VLM-based planner SGDrive by 0.4 PDMS and the world-model planner ExploreVLA by 1.1 PDMS under the same sensor setup. Among world-model-based planners, it achieves the highest driving area compliance and ego progress while maintaining a competitive no-collision rate and time to collision.

Starting from an action-only model that scores 86.6 PDMS, adding video co-training lifts every metric and boosts PDMS to 90.3 by imparting traffic-dynamics priors. Subsequent reinforcement learning further raises PDMS to 91.5 by better balancing progress against safety, yielding a combined improvement of 4.9 points. Action-only model achieves 86.6 PDMS. Adding video co-training improves all metrics and raises PDMS to 90.3. Reinforcement learning pushes PDMS to 91.5, largely by raising the progress score (EP) while time-to-collision (TTC) and compliance (NC) dip slightly. Video co-training and RL together contribute 4.9 PDMS points of improvement.

The isolated attention mask attains the highest planning score while decoupling the action expert from future video tokens, unlike bidirectional and action-to-video masks that tie the video branch to inference. It achieves the best NC and TTC with only a slight trade-off in EP, showing that exposing the action branch to future video tokens provides no measurable benefit and that a simpler dependency structure enables efficient deployment. Isolated attention yields a PDMS of 90.3, outperforming bidirectional (90.2) and action→video (90.1). The isolated mask records the strongest NC (98.7) and TTC (95.9), while EP is slightly lower at 83.9. Bidirectional and action→video masks couple action prediction with future video tokens, making the video branch mandatory at inference yet delivering lower planning scores than the isolated design. Decoupling the action expert from future video removes deployment dependencies without sacrificing overall driving quality.

SimWAM’s unified attention interface works with a range of pretrained video backbones, with PDMS ranging from 88.7 to 90.4. Backbones pretrained on driving data, like Cosmos-Predict2.5, provide stronger dynamics priors and achieve the best overall planning score, while a lightweight backbone (LTX-Video) clearly trails. The results show the method is not tied to a specific video model and can readily absorb higher-quality, domain-relevant video priors. Swapping video backbones yields PDMS scores between 88.7 and 90.4, demonstrating robustness to the choice of pretrained generator. The driving-pretrained Cosmos-Predict2.5 achieves the highest PDMS (90.4) together with top EP and TTC, confirming that domain-relevant priors improve planning. The lightweight LTX-Video backbone attains only 88.7 PDMS, indicating that video prior quality significantly influences performance. Wan2.1-1.3B and Wan2.2-5B achieve similarly strong PDMS (90.2 and 90.3), showing the framework is not tied to a single backbone.

Scaling the action expert from 0.21B to 1.02B parameters steadily increases the driving score from 89.9 to 90.3 PDMS, with consistent gains in safety and compliance metrics. The independent two-expert design allows the action expert to be sized according to planning quality and efficiency needs, while a larger video expert can strengthen training supervision without affecting deployment cost. PDMS rises from 89.9 to 90.3 as the action DiT size grows from 0.21B to 1.02B. NC, DAC, and TTC improve slightly across scaling steps, while EP remains stable, indicating no trade-off in progress. The decoupled architecture enables independent scaling, letting the video expert improve training priors without inference overhead and the action expert balance quality against efficiency.

SimWAM is evaluated on the NAVSIM navtest benchmark using a single front camera, where it achieves state-of-the-art end-to-end planning performance. Ablations show that video co-training and reinforcement learning progressively lift the driving score by injecting dynamics priors and balancing progress with safety, respectively. The model’s isolated attention design decouples the action expert from future video tokens, removing inference-time dependencies without harming driving quality. Additional studies confirm the framework’s generality across different pretrained video backbones and its ability to effectively scale the action expert.