One-sentence Summary

The authors propose VFlash, a fast and accurate visual-planning method that uses Flash Primitive Decoding to jointly predict primitive coordinates with a lightweight drafter generating entire primitives in one forward pass, accepts structurally valid predictions without target-token agreement, reduces prediction errors on 2D and 3D robot trajectory tasks, and achieves roughly 25× faster 3D point motion forecasting, enabling a world action model to reach 70.2% average success across four MolmoSpaces task suites compared with 62.6% for the baseline.

Key Contributions

• VFlash is a fast visual planning method based on Flash Primitive Decoding, using a lightweight drafter that predicts an entire visual primitive jointly in one forward pass and supports compact boxes and long variable-length trajectories.
• The method trains on annotated primitives and directly accepts structurally valid primitive predictions without requiring exact token-sequence agreement with an autoregressive model.
• Across evaluated 2D and 3D robot trajectory tasks, VFlash reduces prediction errors relative to autoregressive decoding. Preliminary single-rollout measurements indicate approximately 25× faster 3D point motion forecasting, and a world action model conditioned on VFlash plans achieves 70.2% average success on four MolmoSpaces task suites versus 62.6% for the baseline.

Introduction

Vision-language models are increasingly used as high-level planners for physical AI systems, producing visual plans such as bounding boxes, trajectories, and dense 3D motion forecasts that make spatial goals explicit before action. This matters for robotics and autonomous driving because these visual plans can condition downstream action models and must update rapidly under latency budgets. However, autoregressive VLMs decode coordinate tokens sequentially, so longer trajectories, additional objects, and joint 2D/3D outputs require more decoding steps; prior parallel methods either trade speed against quality or are limited to fixed-size 2D primitives and token-level verification. The authors introduce VFlash, a generalist decoding framework built on Flash Primitive Decoding. It aligns drafter training and inference with complete visual primitive boundaries, uses a lightweight diffusion drafter conditioned on frozen target VLM features to predict an entire primitive in one forward pass, and accepts structurally valid visual drafts without target-token agreement while retaining verified speculative decoding for text.

Method

The authors present VFlash, a generalist decoding framework designed for physical AI that accelerates the generation of spatial predictions. Building upon the speculative decoding paradigm of DFlash, the method introduces Flash Primitive Decoding to replace token-level verification with the direct acceptance of structurally valid visual primitives.

In traditional speculative decoding, a draft model proposes a sequence of tokens that the target model verifies in parallel. However, token-level agreement does not necessarily reflect spatial accuracy. VFlash retains the drafting architecture but shifts the unit of parallel prediction from arbitrary text tokens to complete visual primitives, such as bounding boxes, robot trajectories, or future point positions. Each primitive is governed by a schema that specifies its type, coordinate dimensions, and extent.

As shown in the figure below:

The framework relies on two core design principles for token representations: boundary alignment and position alignment. Boundary alignment ensures that each draft block covers one complete primitive, avoiding the fragmentation caused by fixed-length windows. A tool call declares the block length and layout, allowing the entire primitive to be drafted in a single bidirectional forward pass. Position alignment assigns a known role to every slot, such as a specific waypoint, axis, or delimiter, before decoding begins. To maintain this fixed-width layout, the authors support both special coordinate tokens and text-tokenized targets. For text-based targets, each coordinate is allocated a fixed number of positions, with unused high-order positions filled by mask tokens. This fixed-width grid preserves position roles and is seamlessly rendered back into the target text format.

For the training process, the authors adapt an autoregressive Vision-Language Model to the spatial response format using next-token supervision. They then freeze the target model and train a shallow, non-causal drafter on its extracted features. Given an annotated block, the training objective minimizes the weighted cross-entropy loss:

$$\mathcal{L}_{\mathrm{prim}} = - \sum_{j = 1}^{B(s) - 1} w_j \log q_{\phi, j} \Big(a_j \mid H_{\theta}(x), a_0, \mathrm{MASK}^{B(s) - 1} \Big)$$Lprim​=−j=1∑B(s)−1​wj​logqϕ,j​(aj​∣Hθ​(x),a0​,MASKB(s)−1)

where $B(s)$B(s) is the block length defined by the schema. All logits are computed jointly in a single forward pass. Crucially, the labels are derived directly from annotated coordinates or rendered grids rather than the autoregressive continuation. This optimizes the drafter for spatial accuracy instead of mere token-level agreement. Uniform weights are applied to primitive positions, and packed attention is utilized to keep independent blocks separate while preserving internal attention.

During inference, the decoding process runs standard DFlash blocks until a primitive-start token is accepted. The decoder then resolves the declared schema and drafts the complete block of length $B(s)$B(s) in one parallel pass. Per-position constraints restrict coordinates and delimiters to their legal vocabularies, and cross-position rules pair out-of-frame coordinates. These constraints act directly on the computed logits without requiring additional neural evaluations. The acceptance test is replaced by a schema validator and, for bounding boxes, a spatial confidence guard. If the primitive is valid, it is committed entirely without token-level agreement, requiring only a single target pass to update the cache and context features. Substituting this direct acceptance mechanism into the latency equation yields:

$$L_{\mathrm{dir}} = \frac{t_{\mathrm{par}} + t_{\mathrm{target}}}{B(s)}$$Ldir​=B(s)tpar​+ttarget​​

This design ensures that the latency per token remains constant regardless of the primitive complexity, as a more detailed plan adds no sequential decoding steps.

Experiment

The experiments evaluate VFlash using a customized embodied VLM with per-task autoregressive targets and a separately trained drafter, while action models condition on the frozen VLM's visual-plan features. Visual primitive evaluations show improved or competitive 2D and 3D trajectory, bounding box, and dense motion prediction, alongside substantial decoding speedups and some tradeoffs on strict or dense metrics. Downstream, visual-plan conditioning improves manipulation success and LIBERO robustness, especially with joint 2D+3D plans, and yields competitive autonomous driving results without updating the planning VLM. Throughput analysis attributes the latency reduction mainly to decoding and shows that token-level verification limits gains for subtask text, whereas block-level drafting of visual primitives is much faster.

The primitive layouts organize visual plan blocks around complete primitives, with shared templates that assign every slot a fixed role before decoding. They cover bounding boxes, 2D and 3D trajectories, and dense motion, using point counts and special tokens to keep coordinate positions consistent. Fixed-width special coordinate tokens avoid text tokenization variability, which supports parallel prediction and per-position constraints. Each primitive block is boundary-aligned, so one block covers a complete primitive rather than a fixed-length window. Special coordinate tokens occupy one position, preventing variable text tokenization from shifting subsequent coordinates.

Open-loop 2D trajectory prediction shows VFlash consistently improves on its autoregressive target across both ShareRobot-Bench and VABench-V. The gains are broad across DFD, HD, and RMSE, and VFlash is especially strong on VABench-V while trailing only RoboBrain-2.0 on ShareRobot-Bench. VFlash reduces all six trajectory errors relative to its AR baseline, with HD improving by about 7% and DFD and RMSE by 3–6%. Compared with prior models, VFlash achieves roughly 40% lower errors on VABench-V and ranks second only to RoboBrain-2.0 on ShareRobot-Bench.

On COCO and Dense200, VFlash with AR fallback closely matches the autoregressive baseline in mean F1 while decoding boxes about 2.6 to 2.8 times faster. The fallback setting improves F1 at IoU 0.50, particularly on dense scenes, but lags the baseline at IoU 0.95. Direct generation without verification is fastest yet produces the weakest F1, with a sharp mean F1 drop on Dense200. VFlash with AR fallback roughly matches the baseline in mean F1 on both datasets. The fallback configuration decodes boxes about 2.6 to 2.8 times faster than the autoregressive baseline. Fallback improves F1 at IoU 0.50, especially on Dense200, but reduces F1 at IoU 0.95. Direct generation without verification is fastest but substantially lower in mean F1 on Dense200.

VFlash improves open-loop 3D trajectory prediction across InternA1, AgiBot, and MolmoBot on all metrics relative to AR decoding under every target variant. Waypoint accuracy improves substantially, and VFlash maintains similar 3D accuracy when jointly predicting 2D and 3D outputs, whereas AR worsens. Adding wrist-camera views during training further reduces VFlash error even though evaluation uses only the primary camera. VFlash lowers ADE and FDE and raises waypoint accuracy at 5 cm and 10 cm across all datasets and target variants. With 2D and 3D outputs, AR 3D ADE worsens by 15 to 51 percent compared with 3D-only outputs, while VFlash keeps similar accuracy. Wrist-camera training further reduces VFlash ADE by 3 to 12 percent without changing single-view evaluation inputs.

For dense 3D point motion forecasting, VFlash drafts query-point future frames with text-tokenized coordinates and uses MolmoMotion-AR as its target. It reduces DAVIS ADE and FDE by 27% and 19%, respectively, and HOT3D FDE by 8%, while ranking second on HOT3D and WorldTrack ADE behind the AR target. The approach is about 25x faster in single-rollout generation, though some tracking metrics lag. VFlash improves DAVIS ADE and FDE by 27% and 19% over its AR target and lowers HOT3D FDE by 8%. VFlash ranks second on HOT3D and WorldTrack ADE, ahead of the flow-matching variant but behind the AR target, while some waypoint tracking metrics are lower.

The experiments evaluate VFlash’s primitive-based layouts on open-loop 2D and 3D trajectory prediction, bounding box generation, and dense point motion forecasting. VFlash consistently improves over its autoregressive target on trajectory benchmarks, closely matches box-generation F1 when using autoregressive fallback while decoding substantially faster, and preserves 3D accuracy under joint 2D and 3D prediction. For dense motion, it improves DAVIS and HOT3D errors but remains behind the autoregressive target on several tracking metrics.